Behind the Scenes: The Technology of the New Spoke

As you know, Spoke recently re-launched as a moderated wiki of business information, and it is quite different from our previous service. We still have many of the same company and person profiles, but how you interact with the information is brand new. We now allow you, the community, to update and edit the information, either as a registered user or anonymously. When we began this project, a little over one year ago, we knew that the end product would be different enough that we would have to build this project from scratch. This posed an important question to be answered: on what technologies do we build the new Spoke web application?

The infrastructure for the previous version of Spoke can be summed quickly as Java, Struts 1.x, MySQL, Microsoft SQL Server, and a home-grown service-oriented architecture (SOA) infrastructure. We also had a codebase that was almost 9 years old. Since the new application had to be built from scratch, we had the opportunity to choose a new platform for development.

Our first decision was to choose a new web framework. We explored a few frameworks such as Scala on Lift, Python on Django, Java on Spring MVC, and Ruby on Rails. Ruby on Rails, in the end, was the clear choice for our team. Despite not having experience with Ruby development, we felt that the framework had the focus on rapid development that we desired. Furthermore, both Ruby and Rails have great online documentation, several books, numerous blogs , and a large, active community.

Coming from a Java background, it took us a little while to get used to developing with the Ruby programming language and the Rails framework. However, after a few weeks we started to reap the benefits of Ruby on Rails and appreciated the ease of development over using Java frameworks like Struts and Hibernate.

There is no arguing that the Java development community is expansive, has plenty of books, good documentation, and several web frameworks. Although we could have chosen Java again, Rails has proven to be excellent for rapid web development while Java has lagged behind in this regard. There are other things that are just easier with Ruby as well. Java has tools such as Maven to help manage dependencies and pull in new libraries, but it does not compare to the ease of using the Ruby gems system. Furthermore, there seems to be a gem (i.e., a Ruby library) for just about everything. Need authentication and login for your website?, There's a gem for that: devise. Authorization? There's a gem for that: cancan. Without such a plethora of Ruby gem libraries, we would not have been able to complete this project as quickly as we did.

After we decided upon a web framework, the next key decision our team had to make was what database to use. We needed a database that is web scalable, easy to maintain, and suited to our read heavy access pattern. For the previous version of Spoke, we were primarily using MS SQL Server and MySQL. We did, however, deploy one feature on the previous Spoke site using MongoDB in order to get a feel for using it in production. The test was a success and drove our decision to deploy the new Spoke site using MongoDB. Other factors in the decision included the built in ability to shard the database (as needed), replication, schema-free database flexibility, support for Ruby on Rails 3 via . Features aside, the fact that many high traffic internet startups, such as IGN and Foursquare are using Mongo DB eased any concerns we had about deploying this NoSQL database in production.

Our next decision was where to host the new site. We're a small team, and the previous incarnation of Spoke is still hosted in a co-location facility on hardware, which we own. When a server goes down, with travel time, it can take several hours to get it back up and running. To scale up, it required a lot of lead-time to order new servers, get them provisioned, and deployed into production. In order to alleviate these issues and to supplement our colo servers on the old Spoke site, we were already using Amazon Web Services EC2 and S3 in order to launch new infrastructure and features. When it came time to decide where and how to host the new Spoke web service, the decision was easy: launch the new site 100% on Amazon Web Services.

Our final decision was how to deal with searching the profiles in our system. The old Spoke search was built upon a very old version of the Lucene library as well as home-grown code to manage indexing, searching, swapping indexes, and more. We looked at open source projects such as Solr and Elastic Search, but as a small team we would rather not have to manage yet another large piece of infrastructure. That is when we found IndexTank, which provides search infrastructure as a service. All we had to do was write code to use their APIs for indexing documents and executing searches , and they handled the infrastructure and scaling. The end result is very fast and very good search in the new Spoke system.

As a result of carefully choosing our technology platform, the launch of the new Spoke web application has been successful. The infrastructure and frameworks that we have chosen have been able to scale and handle the same levels of traffic as the previous Spoke application. We fully expect that the combination of Ruby on Rails and MongoDB hosted on Amazon EC2 will further scale to handle our growth throughout the year and beyond.

I hope you have enjoyed this small peek into our new technology infrastructure. Please feel free to ask any questions you may have about it in the comments section below.

Claim your profile is live

I am proud to announce a new feature on the site: the ability to claim your profile. As people are contributing to the site, this new feature enables them to link their edits to their own Spoke person pages. In the specific case below, Matt will get two links to its own personal page enabling people interested in his contributions to click on his name and access his personal spoke page.

At this point, the ability to link your page to a page you edited by claiming your profile is only possible when:

1)  You create or edit a page. Right now, in the upper right side of the page, we feature who created the page as well as who edited the page last, giving people incentive to add more content;

2)  You add a video, or a link to the timeline, notable links, or achievements and recognition section.

We tried to make "Claiming your profile" as simple as "following a profile". You get on your profile page and click on "this is me" on the right side. If you want to unlink, you just hover over the same button and click, and it's done.

Personally, I like this feature a lot because it kills two birds with one stone. It reinforces the quality of the links contributed by providing information about who is making the contribution and this also helps contributors to build their reputation on the site. This is our first foray into enabling our contributors to benefit from the pages they are creating. Their personal page becomes the page where they can promote themselves while the pages they are updating with information remain as objective as possible. Claiming your profile is all about the person making the edits, so that others understands who the person is and learn to trust that person’s edits over time.

In order to make this approach a reality, we made several design decisions that you should be aware of:

1)  You cannot claim your profile unless you have registered using either your Linkedin profile or your Facebook profile. If you have already registered, go on the “edit account” page, and link your user account to your social profile(s) from there. This is a safety mechanism that we are using to make sure that people who are claiming a profile are real and representative of who they say they are on the site;

2)  You can only claim one profile. This was a big subject of discussion in the company and this may evolve over time; but we wanted to make sure that you only claim something of value to you and preferably your own profile. We will particularly pay attention of who claims what and we reserve the right to “unclaim” a profile if the profile was claimed inappropriately;

3)  Lastly, you cannot claim a company or a topic. At this point, claiming is all about creating and adding links, and enabling people to understand who is performing these actions. As we develop the site, we will come up with many ways to associate companies and topics to other objects within the site.

 Hopefully, you will appreciate this feature and will learn to benefit from it.

 As usual, please do not hesitate to give feedback at getsatisfaction.com/spoke

 Philippe Cases

Guidelines for editing information on the site

As we have talked about over the last month now, the new Spoke is now live and really shining. I want to thank the team for all the work they have done and also give you some guidelines on how to edit information on this new site.

We are a business information site, our mission is to inform and not to promote. So we are not a marketing directory or a supplement to your web site and we reserve the rights to remove any content that we deem too marketing oriented and not rich enough in information. For instance, we saw one company that added more than 100 tags. This may be good for the SEO of the page but we believe that it is not in the interest of a user interested in business information. We want you to describe the person or the business in the summary and use tags so that a person interested in the profile you are editing gets a sense of what this company is all about.

In term of tone as well as content, the Spoke pages are places for conversations and we want those conversations to be cordial as if we were having them during a business dinner. Some comments would not be allowed and some people would be asked to leave the table if they were too harsh or disrespectful. We will apply the same type of conduct at Spoke: you may say things that are provocative and controversial but we aim to keep the conversation objective and within the boundary of decency and respect. If somebody doesn't respect this code of conduct, we will ban them from editing on this site.

In the summary or in the tags, we don't want information that could be considered as defamatory or slandering such "this person lost 10% of revenue during his tenure", "so and so is having an affair with XYZ", "this person cooked the books". This type of information could be eventually added to the notable links section as long as there is an article on the Internet from a reputable source to back it up.

Right now, we cannot accept any information from another language. Our spam engine only works in English so we would not be able to decipher whether the content added is appropriate or relevant. We will let you know as soon as we add other languages.

Signing in LinkedIn, Facebook or Twitter or even registering gives you of course more privileges as we are able to validate who you are. It doesn’t mean that all of your edits will be accepted but you would get a great level of trust

We want our users to either update a profile to increase its accuracy or to request remove its profile.  This is the role of our moderation team to enforce this policy:

·        So if you replace a correct name by a “:” or a set of random characters, your edits will be rejected by our moderation team.

·        If you want a profile to be removed, you should click on the following link page for removal that you can find at the bottom of any profile person page. If you decide to delete all the information on the profile, you will first do far more work than you should and our moderation team will reject it. Instead use the request for removal button at the end of the page.

·        You can remove information from a profile if you want to. We, in fact, make it very easy to do so. However, if you are anonymous editors, the best practice is to first add some level of correct information and then remove existing information. If you don't do that then the moderation team doesn't have any way to gauge the seriousness of your stream of edits and may reject all your edits because they can't validate them.

·        If a job is not current anymore, use the past association to express that the job is not current anymore rather than delete the information; this information was correct in the first place so why delete it.

·        If a job is not accurate, just delete it and add another one that is current. If you don't want to add another job, do not delete it as it may be rejected by our moderation team.  You should just indicate that the information is not correct by adding the term inaccurate to the job title in question and our moderation team will take it from there.

Is hub.spoke.com, soon to be www.spoke.com, a social network?

I received yesterday from Elmer Shore the following comment on getsatisfaction:

"If i'm seeing & understanding this site correctly, Spoke would basically be another online profile page to an already many existing social media's out there, to which, i'm still trying to unplug from & no longer be found no more, yet here it is again, don't wanna mispoke about you guys or anything, but im sure your intention is well founded to other competitors. Best of luck & success to you."

I want to thank Elmer for his comment and we are always happy to receive comments whether they are praises, criticisms or like this one requirements for clarification. The easy part of the clarification is that if you are an individual, you can remove yourself from the site by clicking on the request removal at the end of the page.

The second comment is about the nature of hub.spoke.com (soon www.spoke.com) and let me clarify this right away: we are not a social network. We are a platform for you to share knowledge on a company, person or a topic of interest to you the same way you contribute content to Wikipedia. The difference with Wikipedia is that your contributions can be assigned to you publicly and if you contribute, you will be able to promote yourself on the site by claiming your profile which will be linked to your edits. However, it is not mandatory to have a profile in order to participate.

Our aim is to be the authoritative source for business information and to help knowledgeable people get recognized for the quality of their content they provide to the platform. Obviously, we are far from our first goal now but we have a good start and with your help, we are committed to get there in a reasonable amount of time.

Migration is for this week end

Over the past few months, you’ve likely heard about the upcoming release of a brand new spoke.com.  We’re ecstatic to announce that the release is close at hand, as we are planning to release the new site live on www.spoke.com this weekend!

To assure that you don’t lose access to a single feature, we will continue to host the current version of www.spoke.com at its new home on center.spoke.com.  In fact we have already set this up, you can go there right now and use the site in its entirety.  Furthermore, your login email address and password will work exactly the same on center.spoke.com.  We strongly encourage you to go there now and bookmark center.spoke.com in order to find it as handy as ever.

As soon as the release occurs, www.spoke.com will host our newly redesigned site, (which you can see now on hub.spoke.com).  Note that your current email address and password will no longer work on the new www.spoke.com, and you can get ahead of the game by going to hub.spoke.com now and registering (which can be done in a matter of seconds using your Facebook, Twitter, or LInkedIn credentials if you prefer).

Please note that during the migration process, both sites may experience brief periods of inaccessibility.  It’s unlikely, but please be patient and wait a minute if you notice this.

We could not be more excited about the pending release.  Keep an eye out for more updates -- we will do everything we can to make the transition as smooth as possible.

As always,  we love hearing from you.  If you have questions or feedback on the current www.spoke.com, please visit our uservoice site; and if you have questions or feedback on the new platform, hub.spoke.com, please visit our GetSatisfaction site!

Thank you again for all your support.

Migration of www.spoke to center.spoke.com

As we are moving forward on our migration path, we wanted to alert you that we have now set up center.spoke.com now for your use. http://center.spoke.com is the exact replica of www.spoke.com. It has all the features that you have available now on www.spoke.com. Your login credentials will work exactly the same for this site.

We strongly encourage you to bookmark the page center.spoke.com in order to find it handily as fairly soon, www.spoke.com will be the home of our new redesigned site, which you can find now on hub.spoke.com.

Please note also that when hub.spoke.com will have migrated to www.spoke.com, you will to have to register on www.spoke.com as the two sites are not backward compatible. We will work over time to integrate the features of the two sites.

If you need any help, just let us know through the regular support mechanism on center.spoke.com or leave us a message on our feedback page http://spoke.uservoice.com/forums/5745-general.