Building an operational data store for an airline
Qantas is the Australian flag carrier airline, carrying nearly 50 million passengers a year. Qantas is currently embarking on some very exciting projects – many of which require real-time data.
Carrying nearly 50 million passengers a year
Key to being able to use real-time data is an operational data store – a place where all data critical to keeping the business running lives; in this instance, bookings and flights data.
Our background in Business Intelligence and Analytics in the airline sector, combined with our strong knowledge of big data tools meant that we could assist Qantas in delivering an ODS. One key aspect of the project was to demonstrate that a few, highly-skilled individuals could deliver very significant value to the business of a short space of time.
Using a combination of Hadoop, MySQL, Kafka and Spark Streaming, we were able to build a platform which processed booking events (seat bookings, passenger changes, upgrades, cancellations etc) in real-time, and made the data available to the rest of the organisation in a handful of seconds after the event occurred.
This was delivered on a highly-scalable and highly-available Amazon Web Services platform, by a team of 4 over 11 weeks. Part of the project involved working with Qantas’ IT, project management, architecture and BI teams to work on evaluating and testing different technologies to ensure they would support varied requirements.
Qantas is now able to process many millions of events per day and make these available to as many consuming applications as necessary, rather than waiting days or weeks.
Standardise data processing
The key to the platform is that interpretation of an event only occurs once, so downstream systems and analysts can focus on interpreting the event, rather than just wrangling the data.
Complex event processing tools can be integrated to the platform to allow Qantas to really tailor how they serve and sell to customers.