Council of Residential Specialists

Project Goals

The main goal of the project was to replace the old and outdated website which interfaced with a complex external member management system that had been built by CRS over several years to maintain a variety of member data.

The client wanted to continue using their legacy system even after the main site moved to Drupal because of its business processes. Functionality from the old site needed to be migrated to the new Drupal site, including member profiles, in addition to new features that would be built and which leveraged Drupal and its capabilities. It was also necessary to perform migration of other data as well as to implement an integration between Drupal and the external member management system that CRS wished to continue using. Data was to be maintained in both places and synchronized, with Drupal serving as the “master” and sending updates to the other system as necessary.

During the development phase we faced a variety of challenging issues. This required a great team effort to deliver the best solutions and workflow adjustments for the client.

Workflow and Scalable Drupal Integration

Initially, the greatest challenge was fully understanding the complexity of the CRS workflow and defining a method for integrating the existing user management system with Drupal. One of the unexpected advantages of the project was that it forced the client to re-examine its own processes and workflows, and this led to a number of ways in which together we were able to improve and streamline what they do. The result was a more efficient and lean way of operating not only the website but their business in general.

In addition, at the time the project kicked off, CP was just starting to use Drupal 7, which had only been released a few months earlier. While many contributed modules had Drupal 7 versions, a number were still unstable or pre-release. For example, modules such as Print, RealName and Views had bugs that took some time to diagnose and track down. Although this resulted in extra work during the development phase it enabled CommonPlaces to contribute to future stable releases of those modules through patches and testing.

Membership Integration and Management

As part of the implementation, it was necessary to integrate the Drupal site with CRS's external member management system. CommonPlaces worked with CRS to identify the data that needed to be exchanged between the two systems and architected custom API and interface modules. The API and interface modules include function calls and parameters that were defined as a result of this analysis. The API module standardizes calls made by the custom Drupal modules that were created by CommonPlaces to manage tasks shared by the two systems, such as membership, user profiles, and course registration. This ensures that if the interface changes at some point, the rest of the site does not need to change. All direct communications between Drupal and the external system are restricted to the interface module, which also handles logging to Drupal's watchdog/dblog and makes debugging easier.

Integration was an ongoing task throughout the project, consuming about six man-weeks between CRS and CommonPlaces. A lot of testing was required to ensure that each component of the integration aligned with the business processes that CRS already had in place, and in some cases, those processes had to be refined or even changed. This resulted in further code changes on the Drupal side to accommodate API changes in the external system.

There were almost 300,000 users that had to be imported into Drupal, and each user profile had a large number of fields representing a wide variety of information being stored and tracked by CRS. In order to accomplish the import, CommonPlaces turned to the Drupal Batch API. Using the Batch API it was possible to import users in chunks of 1000 at a time, and also to retry failed imports without having to empty the system of successfully imported users. Some of the user information was only contained in a single location within the legacy system. Now, however, this information exists across multiple entities in Drupal.

Taxonomy is one example of this. To avoid expensive lookups for references to taxonomy entities when creating each user, CommonPlaces created static mappings of all the legacy data to the unique identifiers of the new Drupal taxonomy.

The timeline for user migration was approximately one month, and like integration, the majority of that time was spent working closely with CRS in an iterative testing and bug fixing process. One of the main issues was inconsistencies in the legacy data being imported into Drupal. For example, many fields required in the new system were not required in the old one. This was resolved with a combination of data cleansing/modification on the CRS side and import script changes to ensure required fields had an appropriate value.

Geolocation, Geospatial Search and Solr Integration

One of the key requirements was a “Find a CRS” tool for visitors to search and easily locate certified realtors within their area or a given area. This would enable consumers and members to drive business to the CRS membership.

The first challenge was generating geolocation data for each of the site’s 300,000 users, based on address. These users also needed to be imported into Drupal, as mentioned above. But the API calls to an external service to get geolocations were slow, and made the import process take much longer than expected. To speed things up we split the geolocation into a separate process that queued up addresses and made use of both that service's batching mechanism and the Drupal Batch API.

Next, there had to be a way to search this data. At the time, Search API was in its D7 infancy and did not support geospatial search, so CommonPlaces built a custom geospatial search.

With almost 300,000 users in the database, and the need to allow searching of user data by facets as well as drilling down, the site needed a robust solution. Solr provides an extremely fast and scalable method of performing these complex searches.

E-Commerce

E-commerce is a major component of the project. The site makes possible the selling of products and membership plans as well as charges for renewal and reinstatement fees. The flexibility of Drupal Commerce proved to be invaluable. CommonPlaces was able to customize various aspects of Drupal Commerce in order to accommodate both the workflow of CRS’s financial department and also data coming from the integration with the external member management system.

However, the access control implementation for purchased resources (course material nodes) could not be performed by Drupal’s built-in permissions system. The improved Drupal 7 node_access hook was not an option since the resources section is so heavily integrated with Views, which will not respect code in a node_access hook for performance reasons. CommonPlaces was already using the ACL module in the forum section of the site, and our decision was to use it here as well to maintain lists of users who had access to certain resources rather than reinvent the wheel with our own node grants-based implementation.

Education and E-Learning

CRS wanted to make sure its members had resources to continue and enhance their learning experience. They also wanted to expand their member base by offering users the ability to gain a CRS Designation. To enable this it was essential to include online courses and e-learning processes and functionality—features that in turn made membership that much more attractive to the client’s target audience.

Through online courses, self-study courses, and webinars, users and members are able to gain in depth knowledge from a wide range of realty-related subjects. They are able to find and select courses based on format, availability, topic, credit type, duration, location, instructor, date, sponsor, and of course keyword search.

Each course has its own access control list which a user is either added to or removed from when they make a purchase, or are granted a refund. Likewise, when a resource node is added to or removed from a course node via node reference, it is added or removed from that course’s ACL. This approach allows a lot of flexibility when managing courses and resources, and since the ACL module uses Drupal’s node grants system, access is respected in lists of content such as Views.

General Approach

There were some very specific considerations that had to be addressed during the development of this project.

The first perspective came from the CRS team. Their main goal was to create a fully integrated product that could make it easier for their members to interact with. At the same time, this integration had to take into account the workflow and processes at CRS.

The second perspective, and certainly the most important, was that of the members and CRS Designees. CommonPlaces had to take into account the importance the members placed on accessing resources and information as well as being able to pay for courses. In addition to this, members needed access to a full social forum to exchange ideas and to network.

The third perspective was that of site visitors interested in hiring a realtor who is a member of the Council of Residential Specialists. The experience and engagement of typical consumers had to be easy and rewarding. Site design and user interface were therefore of particular importance.

The flexibility of Drupal was a crucial factor in the success of this project. The ability to address each perspective enabled us to produce an effective and optimal product that exactly met the needs of the client.

Outcome

Overall, this project was one of the biggest and most robust projects CommonPlaces has so far had the privilege of working on. The client was more than pleased with the outcome, and within the first day was able to deliver new functions and benefits to members and site visitors alike. The fully functioning site has brought Council of Residential Specialists into a new age of membership interaction and contribution while enabling the efficient expansion of the association.