Lessons learned over time

Overfitting a model to the train set in which case it more so remembers the dataset than learning the data-target relation. This results in the model returning gibberish when presented with new data. Use model checkpointing based on the loss of a validation set and check for consistent performance on different subsets of the data.

Classification imbalance occurs when the train set of a classification problem does not have an even distribution of all target classes. Any model trained on this data will have more exposure to a subset of the target labels and in turn over predict this subset. Over- or under-sampling can help.

Fat tails is the regression equivalent of classification imbalance. Many machine learning techniques assume a normally distributed target and fail to adapt well when this is not the case. Quantile regression can help, as can pivoting to a classification problem using quantile classification. Sometimes a log transform can turn a non-normal distribution into one.

Time leaks occur any time data from the future directly or indirectly makes its way into the train set. An example could be using the central difference of a feature, the formula of which uses the value at . This results in a model that, firstly, can't be used in production since it uses data from the future and, secondly, is deceptively good during development.

Last value forecasting is another error that can occur with regression problems. It entails that a model learns to forecast the current value  as the prediction for  since, depending on the problem, it can be a very similar value and thus can achieve a small error. Using the current value as a forecast works until it doesn't but results in graphs that look great with two lines closely following each other. Closer inspection is needed to check model efficacy.

Not using walk forward optimization but cross validation instead. If you simply randomly split and shuffle a time series you run the risk of creating a validation set that basically has duplicate samples with that of the train set. The train set might, for example contain the sample and the validation set the sample . Data at timestamp  is probably very similar to data at  so if you optimize this data split you will get a model that seems really good during development but might not perform well in production.

Don't spend too much time developing models. Instead, your time is better spend finding some review papers detailing the current state of the research field most similar to yours and comparing the best performing models in that field. At most, adapting existing architectures to your specific use case is ok. Developing your own models carries the risk of spending too much time coding and fixing bugs. Time should be spent instead on the data. On data exploration, data cleaning, data preprocessing. Exploring how a data pipeline will behave in production and whether it is viable or not.

Of vital importance is choosing what to predict. Not only should a model be able to learn to predict it, but the prediction needs to be able to be converted into an action. At the end of the day the goal is to convert data, through machine learning, into an action, whatever it may be. It could be a buy, sell or do nothing signal when talking about financial data or it could be to flag or to not flag something for further investigation in the context of anomaly detection. It could be any number of things.
So whatever a model outputs needs to serve this purpose. First working out the prediction-to-action pipeline to check its viability can save a lot of time in the long run. Maybe what you thought to be a useful metric to predict turns out to be inadequate or insufficient.

In applied machine learning, model development should, in my opinion, take a backseat to pipeline development; from data to action. A pipeline could be data ingress to pre-processing to a model to a prediction to post-processing to an action. All steps need to be in place before any reviewing or testing can be done. Prototyping an entire pipeline first and then iteratively improving each step is a more time efficient approach in the long run compared to trying to perfect one aspect first.
In my experience, developing and training a model takes the longest time. So if you first get your data input right, then work on getting a decent model and only afterwards start working on turning predictions into actions, only to learn that whatever you wanted to do to be unfeasible, then you just wasted a lot of time.
Problems you might encounter are speed (the data preprocessing and or model might be too slow) and usability (already discussed in "Target choice matters").

Transfer learning is a powerful thing that can be used quite liberally when working with kernel based models. In case your data has more or less input channels than that of the pretrained model, you can just repeat or drop the weights of the input channels of the model, and it will work just fine. You can also transfer a model from one domain to another. I once transferred a model trained on the CIFAR-10 dataset (trucks, dogs, ...) to a spectrogram dataset and doing so resulted in a better model, faster, compared to training from random initialization.

Analyse the data, the target and the model separately. Data preprocessing should not be fixed to one single model architecture, nor should an architecture be fixed to one target variable. Keep each component modular during development. In Pytorch Lightning this can be achieved by using the LightningDataModule and LightningModule.

I want to get better at deep learning for regression. Specifically, I want to learn how to work with imbalanced target data and how to do extreme value prediction. The issue with the first one being that not all models work well with non-normally distributed data. The challenge with the latter is that extreme values typically occur less often, making a model gravitate towards being good at predicting what does occur more often.

• Currently, the plan is to explore quantile regression and quantile classification further, and explore alternate error metrics like Huber error and squared error relevance area.
• I wonder whether using a second output target can help with regularization. So instead of only forecasting future returns also forecast future variance. This might encourage the model to be a bit more risk averse.
• A sensitivity analysis of kernel length, dilation, stride, causality and depth with TCN. (And seeding?)

First off, there is the initial dataset. Make sure you understand how the data is aligned, especially when data is summarized per window. Is the timestamp identifying the window that of the start, middle or end of said window? Align all datasets the to the same standard.
Data storage also used to be a point of concern but now I just always use parquet files for offline use and QuestDb or just in memory caching if I need an online solution.
Time series data is typically not independently and identically distributed (IID) which makes missing data imputation hard. I identify areas where any of the datasets used have missing values, potentially take a safety margin in both directions (past and future) and mask (exclude) those timestamps during model development.

Note that there are techniques to convert time series into images. If you are more familiar with image classification, then this might be an interesting approach. Multiple algorithms exist, but the general gist is to compare all points in time to each other, creating a matrix/image.

As usual, input scaling is needed when working with deep learning, but additionally, stationarity is something that needs to be taken into consideration.

With regard to scaling, there are many options. Just using per feature standard- or robust scaling can work, assuming feature-magnitude is relatively constant over time. If not, frequent retraining might be needed or alternatively some type of online scaling can be used. The easiest form of online scaling is to just scale every input sample/slice into the [0,1] range. Another approach is to do standard scaling but using the mean and standard deviation of past data. Maybe just that of the last input slice, maybe that of all of, for example, last week's data, or that of the last 100 input slices, etc. This approach is easy to implement both in development and production.

When it comes to stationarity there are again multiple options. One, just don't bother, chances are that whatever is happening in each individual input slice is stationary enough. This is very problem dependent of course, so if in doubt, or if needed, stationary can be enforced by taking the first (or maybe even a second) order difference. Another interesting approach is that of using fractional differentiation. With fractional differentiation, a balance is made between keeping some memory (magnitude information) and making it stationary. In practice a weighted sum of the original series can be taken using the weighting scheme:

with being the factor that balances stationarity and memory. A larger d approaches a first order differentiation and results in more stationarity and less memory. A good resource on this is the book "Advances in Financial Machine Learning" by M. Lopez de Prado.

Because time series problems can be quite large and I'm limited to my desktop computer, I do care significantly about fast model training. One plug and play solution that helps greatly is to use AdaBelief as an optimizer. Another is to use a decreasing learning rate that starts quite large. Personally I adapted Pytorch's learning rate scheduler cosine annealing with warm restarts to have a decreasing amplitude over time, something I've shared in this GitHub repo, but other schedulers can work great too. A third tool that unfortunately isn't always available is to use transfer learning, as discussed in concepts. Do use it whenever you can find a pretrained model with an input format similar to yours.

On the usage of models, I have two things to offer. Scale your action in relation to a model's confidence, and use a weighted ensemble of models. Regarding that first point, I must say that this once again depends on the problem itself, but whenever an action (the thing that happens as a result of a model's prediction) can be scaled, scale it in relation to the confidence a model displays. Secondly, there is value in combining the predictions of multiple models. In fact, InceptionTime, the current state of the art in time series classification, uses the average prediction of 5 models. An equal average of similar models should be taken, but when combining different model architectures it might pay to use an uneven weighting scheme where one model is valued more than another, e.g., if regime changes are expected.

Over the years, I have written a lot of code. Mostly the code needed to process data and to train models, but also code that ran 24/7. To this end, I got better at:

I both have experience writing production code that can run 24/7 and writing the code needed for data analysis and model training.

I might have had a desire to apply deep learning to time series, but the fact of the matter was that my python skills were limited and that I did not know how the financial markets actually work. Needless to say that some studying was needed. I learned to learn from the internet using YouTube tutorials, blog posts, dedicated educational websites, academic papers, and just plain old trial and error, a slow but effective tutor.
Critically, I learned to be critical when it comes to interpreting the results of studies on market data as there are many pitfalls which can make models seem better than they are. In short, the old adage that if something is too good to be true it probably is, rings true. Academic papers and amateur blogs tend to overinflate their results, unknowingly or not.

I am good at familiarizing myself with new domains, researching techniques that might fit that domain and being critical of said techniques.

I am proficient in extracting meaningful insights from complex datasets.

The best way to analyse a model is to run it live. Unfortunately this is not always feasible nor a good idea.

Offline, the classic cross validation or walk forward validation care useful. Training and error metrics are also good tools to get a macro idea of model's performance. Confusion matrices and scatter plots are great ways to get a visual representation of model performance and typically also a good starting point to explore edge cases. Think false positive and false negatives, extreme predictions that are missed or average samples that are overpredicted.
Just make sure not to "mentally overfit" to the data. Be careful not to develop a bias of what a model should do and then finetune said model to that end.

To evaluate online performance a simulation could be used using either offline data being fed incrementally or using live data but in a development environment.

I am proficient in model evaluation, both at the macro and micro level.