The systems that provide stock exchanges and financial clearing houses must never go down and they are sometimes compared to the rigorous scrutiny that control power plants and fighter jets.
One of the companies at the intersection of these inancial IT systems, big data and social media is the New York-based startup company Dataminr.
Dataminr has developed a method that allows information to be retrieved in real time from the huge amount of data that Twitter generates. Diving deep into this data with data mining methods Dataminr is capable finding patterns and structures that can be combined to rapid analyzes of the hot new trends that may affect financial trading. In order to really dig deep into all Twitter updates Dataminr have managed to get a strategic partnership with Twitter, which means that they have direct access to information in the Twitter’s servers in real time.
The method has attracted great interest among banks and financial institutions as an additional tool they can use to take decisions based on movements in financial markets.
In an article on The Next Web Dataminr is hailed as one of the 50 hottest startup companies in New York. The company also attracts investors, $ 30 million has been received this fall in fresh venture capital.
In order to develop the tools and methods that Dataminr use to detect patterns in Twitter updates they need to build up a professional team of computer scientists and mathematicians.
This is evident in the job ads Dataminr publishes. As of this writing Dataminr are looking for ”dedicated data scientists” to sort, analyze and deliver relevant information from the huge amount, we’re talking terabytes, of unstructured data generated by Twitter updates. The formal requirements are a Ph.D. in computer science, mathematics or statistics, and at least five years of experience working with advanced data mining, statistical analysis and machine learning.
The importance of computer scientists in financial IT is stressed also by Tomas Forsberg from the Swedish company Cinnober when he recently at a lunch lecture with students in Computing Science at Umeå University told that computer scientists and software developers are key holders for the entire financial industry.
Cinnober develops solutions for demanding trading and clearing venues. Exchanges and clearinghouse worldwide are their customers from Thailand and Dubai to London, Brazil and other international hot financial metropolises.
“We are constantly working with risk management and stability. Operational safety is important and real-time monitoring is a requirement”, says Tomas Forsberg.
With experience of analysis and risk management in financial trading and development of the systems internationally major exchanges use to manage their large data sets, Tomas Forsberg has a deep understanding of the important role that computer scientists play in financial IT.
“You are those who can answer questions about how the systems will be built”, says Tomas Forsberg supporting the students to take their first steps into the realm of financial it.
Mattias Lidman is one of many talented computer scientists who have graduated from Umeå University. Today, he is researcher at the Computation Institute of Chicago. In his student thesis, he addressed the same question as Dataminr – is it possible that from the surging Twitter stream do analyzes of trends in the financial market and share trading?
It is, showed Mattias Lidman in his thesis ”Social media as a Leading Indicator of Markets and Predictor of Voting Patterns.” In this paper he shows how the flow of information from social media, primarily Twitter, can be used to predict market trends and to predict how the price of the shares will rise or fall.
To get the most accurate picture of reality as possible Mattias Lidman chose to take on the entire Twitter stream where retweets and spam filtered away. The result was based on unique hits written by people who have an opinion and who are familiar with stock trading.
Using a statistical analysis and filtering, Mattias has caught up with posts that mention the stock symbols for example IT companies like Intel, Google and Apple. To handle the Twitter stream required an automated approach. Millions of different correlations between the values in updates and stock symbols make it impossible to follow it manually.
These three examples from Dataminr , Cinnober and researcher Mattias Lidman indicates the importance of computer scientists today for financial IT and financial trading. It also gives a hint to universities that financial it is a quest also for education in computer science.