IEEE-CIS Technical Challenge on Energy Prediction from Smart Meter Data

Submission Dates:
08/15/2020 to 11/15/2020
Citation Author(s):
Isaac
Triguero
University of Nottingham
Submitted by:
Isaac Triguero
Last updated:
Tue, 07/20/2021 - 06:35
DOI:
10.21227/2npg-c280
Data Format:
License:
Creative Commons Attribution

Abstract 

Imagine you just moved to your brand-new home and hired your energy provider. They tell you that based on the provided information they will set up a direct debit of €50/month. However, at the end of the year, that prediction was not quite accurate, and you end up paying a settlement amount of €300, or if you are lucky, they give you back some money. Either way, you will probably be disappointed with your energy provider and might consider moving on to another one. Predicting energy consumption is currently a key challenge for the energy industry as a whole. Predicting the consumption in a certain area is massively complicated due to the sudden changes in the way that energy is being consumed and generated at the current point in time. However, this prediction becomes extremely necessary to minimize costs and to enable adjusting (automatically) the production of energy and better balance the load between different energy sources. Smart meters are being rolled-out in many countries for domestic use, becoming powerful devices to track energy use. Smart meters will not only play an increasingly large role in the way customers consume energy but also in the way they choose a supplier. If energy providers developed new products for their customers that help them understand the potential benefits of smart meters (e.g. reducing bills or managing their finances better), this could be key to attracting and retaining customers. At a higher level, being able to use smart meter data to better manage demand will contribute to the increasing use of sustainable sources of power and reduce the demands on more traditional power generation. Researchers from the IEEE Computational Intelligence Society (IEEE-CIS) want to improve energy prediction based on Smart meter data, while also improving the customer experience. IEEE-CIS works across a variety of Artificial Intelligence and machine learning areas, including deep neural networks, fuzzy systems, evolutionary computation, and swarm intelligence. Today they’re partnering with the leading international energy provider, E.ON UK plc., seeking the best solutions for energy prediction using Smart meters, and now you are invited to join the challenge. From a machine learning point of view, the provided data is very challenging and may lead to the development of novel learning approaches. Some of the challenges include incomplete data (i.e. missing values), use of external data sources to handle seasonal effects, different kinds of households (e.g. families vs. single, old house vs. new built), or lack of sufficient information about the households. If successful, you will not only help many to automatically control their energy bills, but the proposed technical challenge may be applicable in many other similar fields facing time-series prediction problems. Please report any issues or feedback to Isaac.Triguero@nottingham.ac.uk

Instructions: 

Data description and submission requirements

IMPORTANT: Data has been updated on September 7th, 2020 after finding a bug in the timestamps.

The goal of this competition is to predict the monthly electricity consumption for 3248 households in a coming year (January to December). You are provided with historical half-hourly energy readings for the 3248 smart meters. To simulate a realistic use case, we take the 1st of January of a given year as the day we want to make predictions. Thus, different smart-meters will have available a range of months’ worth of consumption, ranging from only last month (i.e. December) to the entire last year (January to December), acknowledging that customers may have joined at different times during the previous year. For example, we may have a few customers for which we only have data from last December, and we aim to predict January to December of the coming year, whilst for others, we may have the entire January-to-December time series. We would like to see how well we can predict the coming year depending on the amount of data that is available. 

For each meter_id in the provided dataset, you must predict the consumption in the following 12 months in kWh. In addition, you are requested to submit a draft description of your methodology (up to 1000 words). Note that If you are working in a team you should indicate this clearly in your final submission. Only one submission for each team will be considered during the shortlisting.

Full description of the data and submission requirements can be downloaded here

Evaluation 

Our partner E.ON is interested in predicting with the least amount of data as possible a good estimate of the total year consumption for reasons of payment adequacy. In addition to that, an accurate monthly consumption prediction is also very valuable for Energy Trading teams, who need to be able to predict with some accuracy how much electricity to buy on the energy market. We will use the relative absolute error for monthly and yearly predictions. See the Leaderboard and all submissions at the bottom of the page or alternatively here.

Details about the evaluation can be found here

Note that the Leader Board table is updated every 5 minutes. In case you submit a wrongly formatted submission, you will be listed at the top of the submissions' table with an empty row.

At the end of the competition, and looking at the aspects explained above, the Technical and Scientific Committees will shortlist the top 5 submissions. Shortlisted authors will be asked to provide a final description of their methodology (4 pages in IEEE format, more details will be provided at the time).

Final submissions will be carefully assessed according to the following criteria:

-        Performance in different scenarios, including annual and monthly predictions, as well as predictions with limited historical data of the user.

-        Novelty of the proposed approach and appropriate use of Computational Intelligence techniques if any (not required!). The Scientific committee will be asked to rank the shortlisted independently and this will be used to compute a score.

 Note that use of data other than the one provided is not allowed. If you do so, we will not consider your submission.

Timeline

-        August 15th, 2020 – Competition start.

-        November 15th, 2020 – Final submission deadline.

-        November 18th, 2020 – Shortlisting announcement.  

-        November 25th, 2020 – Deadline final description report.

-        December 1st, 2020 – Shortlisted solution presentations at the 2020 IEEE Symposium Series on Computational Intelligence (SSCI). The conference will be held virtually and registration for the 5 shortlisted submissions will be covered.

-        December 4th, 2020 – Awards ceremony

All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organisers reserve the right to update the contest timeline if they deem it necessary.

 

Prizes

Awarded by the Committee based on the criteria mentioned above.

-        1st Prize: $7,000

-        2nd Prize: $5,000

-        3rd Prize: $3,000

-     4th and 5th Prizes: $1,000

All teams, regardless of place, are also strongly encouraged and invited to publish a manuscript of their solution (and open source their code, if possible).

 

Results

After the review process from the Scientific Committee, the Technical Challenge Committee of the 2nd Technical Challenge of the IEEE Computational Intelligence Society is delighted to announce the final ranking and awards of this exciting competition on energy prediction from Smart Meter Data!

 

This final ranking looks as follows:

- 1st position: Wenlong Wu - $7000

- 2nd position: Steffen Limmer - $5000

- 3rd position: Jesus Lago - $3000

 

Runners-up:

- Kasun Bandara, Hansika Hewamalage and Rakshitha Godahewa - $1000

- Sven Rebhan and Nils Einecke - $1000

- Alexander Dokumentov and Fedor Dokumentov - $1000

 

Technical Challenge Committee

•       Isaac Triguero (Chair, Associate Professor, University of Nottingham)

•       Catherine Huang (Principal Engineer, McAfee LLC)

•       Hussein Abbass (Professor, University of New South Wales)

•       Juan Bernabé-Moreno (Chief Data Officer E.ON SE)

•       Luis Magdalena (CIS VP TA, Universidad Politécnica de Madrid)

•       Manuel Roveri (Professor, Politecnico di Milano)

Scientific Committee

  • Robert Eigenmann, E.ON
  • Steffan Birr, E.ON
  • Alberto Fernandez, University of Granada
  • Ricardo Cerri, Federal University of São Carlos
  • Daniel Peralta, Ghent University
  • Jaume Bacardit, University of Newcastle
  • Alejandro Rosales, CIMAT, Mexico.
  • Mikel Galar, Public University of Navarre
  • Natalia Rodriguez, ENSTA Paris
  • Alberto Cano, Virginia Commonwealth University

Acknowledgements

E.ON SE provided the dataset for this competition. E.ON is an international, privately owned energy supplier based in Essen, Germany. With a clear focus on two strong core businesses E.ON aim to become the partner of choice for energy and customer solutions. E.ON provide solutions for the new energy world.

The Chairs of the IEEE-CIS Technical Challenge Committee would like to extend our thanks to:

-      Dr Robert Eigenmann, Nicholas Harbour and Dr Stefan Birr from the E.ON team.

-        Selvi Ergen, KTP associate, funded by E.ON and Innovate UK under KTP project 11094.

-        Heda Song, PhD student and Researcher at the University of Nottingham.

Leaderboard (only best submission for each participant is shown)

All submissions

Comments

Dear participants,

Unfortunately, we have found a mistake in the data we uploaded initially, and I have just uploaded an updated version.

Please accept our apologies for this.

Thanks to Dr Jessa Bekker and her team for finding this.

Best wishes,
Isaac

Hi,

I am not able to download the Data.zip. Could you please check?

Regards
Anjanita

Dear all,

Someone asked about how we handled the potential missing values in the test set, and we didn't specify that above. So, just so you all are aware, the missing values in the test set (if any) were imputed using a simple linear interpolation on a daily level.

Best wishes,
Isaac

I removed the old files from an analysis, but I cannot upload the new files and the following message appears in a green box: "The trimmed version of your post shows what your post looks like when promoted to the main page or when exported for syndication. You can insert the delimiter "" (without the quotes) to fine-tune where your post gets split." I don't know what this means. I click on upload and after a time appears 'No file selected'.

Best regards,
Jeanne

I just changed the names of the files and this worked well for me.

Dear Technical Challenge Committee,

Some time ago the leader was:
Baseline 16 Sven Rebhan 0.2939 1.0595 0.6767 2020-11-05 10:40:31

Now the leader is:
data predictor 1.6 Nils Einecke 0.2934 1.0632 0.6783 2020-11-09 22:55:55

Where is Sven Rebhan's result (which is better)? Are these two people working as a team?

Maybe I am confusing this competition with Kaggle...

Thanks.

Best regards,
Alexander Dokumentov

If you update your submission (e.g, Sven updated Baseline 16), and the new result is worse than before that would be reflected in the leaderboard.

I do not display a leaderboard with the best historial results for each, but the current best result for each user.

I don't know if Sven and Nils are working together, but if they are, they will have to say it in the final submission; therefore, only one submission would be considered during the shortlisting process. Unfortunately, the leaderboard only shows individuals, not teams.

Hope that makes sense.
Isaac

Dear Alexander Dokumentov,

Yes, I'm currently working together with Sven Rebhan in order to try to merge our two approaches. We will, of course, only keep on variant for the final result. We will also write an email to the organizers to state that we collaborate. Unfortunately, we cannot completely remove submission. We will upload random data or older much worse data for all the other submission so that we do not spam the top ranking.

Best regards,
Nils Einecke

Thank you, Nils, for the clarification. I was almost sure that you and Sven were working together. Nice to meet you online. Good luck with the competition, it looks great for you now!

Hey Alexander Dokumentov,

sorry for the late reply!

My contribution vanished because, while trying to improve, I uploaded worse results (I want that month rAE < 1.0 :-))
with the same name and thus appear down in the ranking. You can see this when looking at the submission page and sort
by submission date.
Furthermore, as Nils Einecke already stated we are indeed working on merging our approaches. To our knowledge this is
not forbidden according to the competition rules. Furthermore, for full disclosure I already informed Mr. Triguero that
we are from the same institute. In the interest of a fair competition, Nils and myself will only keep a single
submission together and already stated in the authors list that we are working together for the corresponding
submissions.

If you have any concerns, please let us know!

Best regards,

Sven

Update: I "removed" my submissions (at least those on the first page of the leaderboard) by submitting all-zero predictions.
As we are still trying different merge strategies it might be possible that both Nils and myself will appear on the leaderboard. But again, we will remove all submissions but one for the final ranking.

Thank you, Sven. All clear now. It is good you have a team, I am a bit envy :) Good luck with the competition, your results look great!

Hi, I am trying to submit a new result set using one of my earlier submission entries.
The edit and upload was successful but still the results are not getting updated on the Leaderboard.
Can you please fix the issue?

Hi,

Please make sure that your csv has been correctly uploaded to the platform, otherwise, the leaderboard will not be updated.

Best wishes,
Isaac

Is code sharing allowed between different teams/researchers in this competition? For example, will you allow the first and the second places in the competition to have similar code/similar ideas implemented in their final submission?

No, that wouldn't be allowed, unless you become a single team.

Is it mentioned in the rules? If not, would it be possible to mention it somewhere in the rules section? Thanks.

Thank you for your feedback, Alexander. I have clarified this in the description. Unfortunately, DataPort didn't allow us to create groups/teams easily, but I will suggest this to the platform for the future.

Thank you, Isaac.

Can we please get back the "delete" functionality for submissions. I now have almost a dozen of files flying around to remove my submissions... More to come. I just want to delete that old stuff. :-)

Dear all,

The deadline is approaching and this is just a gentle reminder to update the description of your methodology (up to 1000 words) in your best submission. Note that If you are working in a team you should indicate this clearly in that description. Only one submission for each team will be considered during the shortlisting.

Without this information, we won't be able to consider your submission.

Please provide as much information as possible within the 1000 words to understand your approach.

Best wishes,
Isaac

Dear Technical Challenge Committee,

It looks like the website is unavailable every day after 1 am and then for some time (I am in a different time zone and your 1 am is midday for me). Is it because of some scheduled maintenance? If it is the case, it would be great if you advised what period of time exactly the site is unavailable so that participants do not waste time on submitting solutions during this period.

Best wishes,
Alex

Dear Alex,

I haven't been informed of any scheduled maintenance on IEEE DataPort. I do apologise for this, I will check with dataPort to see what's going on, but unfortunately beyond my control.

Best wishes,
Isaac

Thank you, Isaac.

Same today. 1:40 am GMT and my submission does not appear on the "All submissions" board. Yesterday the system started working at around 3 am GMT.
It would be much less problem if we knew about these outages beforehand.

The submissions are still not working at 3:50 am GMT. It is a bit sad as today is the last day to submit the results.

I consulted this with my contact at IEEE Dataport and they didn't know about this maintenance either.

Hi,
I made a submission on 11/13/2020, however that is in incorrect format. I've updated the format and trying to create a new submission, however the form doesn't accept new files. It says uploading, however the uploaded file's name doesn't appear on the page with "Remove" button.

Also, is there a way I can delete my previous submission or update the result file there itself? Thanks!

Hi Kaustubh,

I have experienced a similar issue with the platform in the past, and it was normally due to not pressing 'Upload' right after selecting the file, and clicking directly on submit. Please check if this works for you. Unfortunately this is something I personally can't improve.

Best wishes,
Isaac

Dear all,

The competition has now closed and I only wanted to say a big thank you to all of you for your participation. We will announce very soon the shortlisted submissions.

With kind regards,
Isaac

Dear Technical Challenge Committee,

as the competition is now closed, I would very much appreciate if you can share the Test Set used for competition results evaluation, just to continue working on this interesting use case and improving obtained results for personal interest.

So the question is: can you share the test set?

Thank you,
Luca

Thank you very much for organizing this interesting contest and providing the datasets!

Indeed, we would be very grateful if you could share the 2018 test data :)

With kind regards,
Konstantinos Theodorakos

Hi Luca, Konstantinos,

Yes, we will release this data, but at the moment we are considering a follow-up competition which will start in a month's time or so.

I will let everybody know more when we make a final decision!

Best
Isaac

Hi Isaac,

According to the competition timeline "November 18th, 2020 – Shortlisting announcement", do you have an update on the shortlisting announcement?

Thanks,
Waylon

Dear all,

It is my pleasure to announce that the Technical Committee has shortlisted the following teams:

Wenlong Wu
Sven Rebhan and Nils Einecke
Alexander Dokumentov
Steffen Limmer
Jesus Lago
Kasun Bandara

Thank you all for your participation. We will post the final ranking after the event at the SSCI conference.

Best wishes
Isaac

After the review process from the Scientific Committee, the Technical Challenge Committee of the 2nd Technical Challenge of the IEEE Computational Intelligence Society is delighted to announce the final ranking and awards of this exciting competition on energy prediction from Smart Meter Data!

This final ranking looks as follows:

- 1st position: Wenlong Wu - $7000
- 2nd position: Steffen Limmer - $5000
- 3rd position: Jesus Lago - $3000

Runners-up:

- Kasun Bandara, Hansika Hewamalage and Rakshitha Godahewa - $1000
- Sven Rebhan and Nils Einecke - $1000
- Alexander Dokumentov and Fedor Dokumentov - $1000

Congratulations to the 6 winners!!

With kind regards,
The Technical Challenge Committee
Isaac Triguero (Chair, Associate Professor, University of Nottingham)
Luis Magdalena (CIS VP TA, Universidad Politécnica de Madrid)
Catherine Huang (Principal Engineer, McAfee LLC)
Hussein Abbass (Professor, University of New South Wales)
Juan Bernabé-Moreno (Chief Data Officer E.ON SE)
Manuel Roveri (Professor, Politecnico di Milano)