Created by Aseem Bansal | Blog Home
Anything new brings along with it many unknowns that we discover with time. After spending some time with any new technology we have a list of things we wish we could send back to our old selves. This is one of those lists that I wish I could send back to our team before we set out to build our first Machine Learning pipeline. Having this could have saved a lot of our time and hope that by sharing this it will save yours.
As I have already shared in one of my earlier posts we were doing things with Apache Spark. So this will contain some general advice for building a machine learning pipeline and other things specific to Apache Spark.
Like anything big and unknown it was hard to do a time estimate. We had an idea about what we would need to do to achieve our goal. But those ideas turned out to be wrong one by one as time went by. We had to accept that and make sure that we can iterate quickly.
There are going to be big unknowns. Ensure that your team is able to iterate.
We have been collecting raw data for around 3 years when we started with the ML pipeline. We were pushing this raw data to our analytics store to turn it into aggregated data. We had not used the raw data for anything but were keeping it in case everything went down and we needed to rebuild our analytic storage. The raw data was in the form of CSV files. We had not noticed that the data had some problems. The problems happened because the code which was writing the files was changed over time so some bugs crept in which went unnoticed. While building the machine learning pipeline we did fix the bugs which were creating the problems in data. We also ended up writing code in Apache spark to do a cleanup of our historical data. The problem was that we found it in the middle of all the things rather than the beginning which increased the difficulty for our team.
Make sure your data is ok before you start.
To train our machine learning models we initially tried loading all of our data. The size of our data was in TBs. We found that trying to load all of it every time made it very slow to train. It also made iterations on improving models slow. We realized that we did not need to load all of the data every time. We were not using all the columns of our data. We did some pre-processing and created a new smaller data set which had the columns we needed for training our models. We also made sure that we didn’t delete the original data source which would serve as a back up in case we messed something up over time.
Don’t mix ETL and model training. If you are training 1000 models you don’t want to do pre-processing 1000 times. Do pre-processing once, save somewhere and then use it for all your model training needs.
As already mentioned we were storing our raw data for backup purposes in AWS S3. We hadn’t really kept it easy to explore from a data science perspective. That was just not the goal when we started dumping data in S3. But when starting work on ML we found that providing easy to explore access to everyone was critical.
Just giving read access wasn’t enough. People can’t just download TBs of data on their laptops, can they? Say someone did download TBs of data then what are they going to do on their laptops? People don’t carry around laptops with 32-cores usually to do processing of TBs of data in a reasonable amount of time. It is just wasting everyone’s time.