Smart metering, Big Data and how to deal with it…

 

By 2020, the Netherlands plans to have 15 million smart gas and electricity meters installed as part of a national rollout following a successful smart meter pilot, according to Dutch Minister of Economic Affairs Henk Kamp.

So, in a few years there will be 15 million smart gas and electricity meters installed, each of these will upload daily the meterreadings taken every 15 minutes (or less) for electricity meters and every hour for gas meters according to the meteringcode. Let’s see how many data records, let alone data, that is assuming there will be 8 million electricity meters and 7 million gas meters:

  • On a daily base: 8 (M meters) x 24 (hours) x 4 (quarters in a hour) = 768 million records for electricity meters and 168 million for gas meters making 936 million records daily
  • On a yearly base: 936 million x 365 (days) = 341 BILLION records

After 3 years, you will have more than a TRILLION records with data! Good luck with putting that in a relational database, it will be very expensive to just store this information into a rdbms. The largest gridco in the Netherlands might own maybe half of all of these meters, but half a trillion is still an awful lot of meterreadings.  Question is, is it really necessary to put this mountain of data into a rdbms?

The rule of thumb I have is: if it is low-value, large quantity of data then don’t store it in an expensive rdbms. If it is high-value, low quantity of data then store it into a rdbms. Anything in between? Pick and choose!

I argue that meterreadings are low-value, large quantity data that shouldn’t be put into a rdbms. Why? because these meters are placed in a consumer connectionpoint, and consumers in the Netherlands typically use 3500KWh each year. Since meterreadings are in KWh, and on average you use 1 KWh each 2.5 hours, so they will only change every 2.5 hours ( 3500/(365x24x4) ). On average only one out of 10 meterreadings will change, the other 9 will not change.

What do I most probably want from meterreadings, since most of them are not changing anyway? I would want to have usage summaries, both per connectionpoint per day/week/month/year and for a collection of connectionpoints.

meterstanden.2

Connectpoint meterreadings summary per connectionpoint (horizontal) and per collection of connectionpoints (vertical)

In order to process this volume of data, one would need simplicity of design, horizontal scaling and finer control over availability. Do these demands sounds familiar to you? They should, because NoSQL databases were built for these purposes. NoSQL databases can easily scale to a great number of cheap nodes, but the price you pay for scalability is simplicity. NoSQL databases can’t process difficult questions like links to other data, but they can aggregate data on a grand scale very fast.

Why not use a NoSQL database as a pre-processing system? Let it do what is does best: fast and cheap processing of simple data. This is what is needed, refining low-value large-quantities of data into high-value low-quantities of data without spending a fortune. Once you have this high-value data, store it into a data warehouse to be processed further.

dwh_architecture.3

NoSQL database as a pre-processor for a data warehouse

By building a number of pre-defined data cubes/reports in a NoSQL database,  you can have a standard stream of data going from the NoSQL database to the data warehouse. For example usage per connectionpoint per month or usage for an entire cityblock could be such a data stream. For a little bit more dynamic refining, a subset of connectionpoints can be transfered from the data warehouse to the NoSQL database. Then the subset can be used as an base population for the data processing.

Personally I find a lot of interesting questions for gridco’s that could be solved this way, like a cityblock KPI dashboard, indicating the health of every cityblock energy usage in a state or nation. Health indicators could be the number of solar panels in a cityblock, and how much energy was delivered back into the network by these solar panels.

image

Example of a KPI Dashboard for power outages

About

122 comments Categories: Big Data Tags: ,

122 thoughts on “Smart metering, Big Data and how to deal with it…

  1. Sweet blog! I found it while browsing on Yahoo News.
    Do you have any suggestions on how to get listed in Yahoo News?

    I’ve been trying for a while but I never seem to get there!

    Many thanks

  2. I’m really loving the theme/design of your weblog. Do you ever run into any browser compatibility issues?

    A few of my blog audience have complained about my website not operating correctly in Explorer
    but looks great in Firefox. Do you have any suggestions to help fix
    this problem?

  3. Hi there! This post could not be written much better!

    Reading through this article reminds me of my previous roommate!

    He always kept preaching about this. I am going to
    send this information to him. Pretty sure he’ll have a good read.
    I appreciate you for sharing!

  4. Hello this is somewhat of off topic but I was wondering if blogs use WYSIWYG editors or if you have to manually code with HTML.
    I’m starting a blog soon but have no coding expertise so I wanted to get advice from someone with experience.
    Any help would be enormously appreciated!

  5. Hey just wanted to give you a quick heads up. The text in your
    post seem to be running off the screen in Chrome. I’m not sure if this is a format issue or something to do
    with internet browser compatibility but I figured I’d post to let you know.

    The style and design look great though! Hope you get the issue fixed soon. Many thanks

  6. Hi, i feel that i noticed you visited my weblog thus i came
    to return the desire?.I’m attempting to in finding issues to improve
    my website!I guess its ok to make use of a few of
    your ideas!!

  7. Fantastic blog you have here but I was curious about if you knew
    of any message boards that cover the same topics talked about here?
    I’d really like to be a part of group where I
    can get opinions from other knowledgeable individuals that share the same interest.
    If you have any suggestions, please let me know.

    Thanks!

  8. First off I would like to say great blog! I had a quick question in which I’d like to ask if you do not mind.
    I was curious to find out how you center yourself and clear your thoughts before writing.
    I’ve had a hard time clearing my thoughts in getting
    my ideas out. I truly do enjoy writing however it just
    seems like the first 10 to 15 minutes tend to be
    wasted simply just trying to figure out how to begin. Any suggestions or tips?
    Thank you!

  9. My brother suggested I might like this website. He
    was entirely right. This put up truly made my day. You cann’t imagine just how so
    much time I had spent for this information! Thank you!

  10. Hi there, i read your blog occasionally and i own a similar one and i was
    just wondering if you get a lot of spam comments?

    If so how do you protect against it, any plugin or anything you can recommend?
    I get so much lately it’s driving me insane so any help is very much appreciated.

  11. Valuable info. Lucky me I discovered your website unintentionally, and I am shocked
    why this coincidence did not came about earlier! I bookmarked it.

  12. Someone essentially help to make significantly posts I might
    state. That is the first time I frequented your web page and thus far?
    I surprised with the analysis you made to make this actual
    put up incredible. Great task!

  13. I think the admin of this web site is truly working hard in support of his website, for the reason that
    here every information is quality based data.

  14. For some, this has changed the idea of the standard vacation.
    Votes and Ingrians were already within the 1300’s sphere of influence inside the Novgorod and
    Moscow beneath the auspices since 1478. There will
    also be bear watching trips, ATV safari’s and trips to Russia and also Finnish sauna and lake
    swimming to experience, golf, birding and fishing.

  15. whoah thiks blog is wonderful i love studying your posts.
    Keep up the great work! You know, a lot oof people are searching around for this info, yyou could aid them greatly.

  16. I’m usually to blogging and i genuinely appreciate your content regularly.
    This written content has truly grabbed my interest.
    I will bookmark your web site as well as continue checking for
    more information.

  17. I desired to thank you a lot more for this incredible website you have made here.
    It’s truly is full of valuable tips for all those who are genuinely
    interested in this subject matter, especially this very post.

  18. Vᥱry great post. I juѕt stumblwd upoon you weblog аnd
    wаnted tto ѕay thɑt Ι’ve гeally enjoyed brwsing yօur weblog posts.
    In аny case I’ll bе subscribing to youг rsss feed and I am hoping yoou writge օnce more very soon!

  19. We are a grop of volunteers аnd strting ɑ neww scheme іn our community.
    Yoour web sire offered սs with valuable info to ѡork on. Yօu have ɗone aan impressive job аnd оur entire community
    wilⅼ be thankful to yоu.

  20. I haᴠe ƅen bowsing online moгᥱ han 4 houurs
    today, yеt I never foᥙnd any interestіng article ljke
    уours. It іs pretty worth enough foг me. In my vіew, іf aall website owners aand bloggers
    mmade ǥood contejt aas yoou ԁіd, the internet wiⅼl bee mᥙch more ᥙseful
    thaan еver before.

  21. I am only commenting to inform you of the amazing experience our girl experienced reading
    the blog. She noticed several pieces, which included
    how like to possess an incredible giving nature to get certain people really easily
    learn certain multifaceted things. You undoubtedly surpassed visitors’ desires.

    I appreciate you for rendering the important, healthy,
    informative in addition to easy suggestions regarding the
    topic.

  22. Ɗo yoou haѵe a spam probⅼᥱm on tɦis blog; I ɑlso am а blogger,
    and I wwas curious ɑbout youir situation; mɑny of us hаve developed ѕome nice procedures аnd we are ⅼooking to swqap methods wth otɦer folks, plezse shoot mе an ᥱ-mail if interested.

  23. I just couldn’t go away your website because I really enjoyed the standard information a person provide for your visitors?
    Is gonna be back regularly in order to inspect new posts

  24. I desired to thank you a lot more for this remarkable website you’ve made
    here. It’s definitely is stuffed with beneficial tips for those who are genuinely interested in this
    area of interest, especially this very post.

  25. Hmm it appears like your website ate my first comment (it
    was super long) so I guess I’ll just sum it up what
    I wrote and say, I’m completely enjoying your
    blog. I as well am an ambitious blogger however I’m still new to everything.
    Do you have any points for newbie blog writers? I’d surely
    appreciate it.

  26. Omg, fantastic blog layout! Just how long have
    you been blogging for? you make blogging seem easy. The overall look of your web site is amazing,
    not to mention the content!

  27. This is a pleasant blog i should say, I usually i don’t post feedback on other sorts of blogs but would like to say that this post truly forced me to do so!

  28. May I simmply jᥙst ѕay ԝhɑt а comfort tо discover someone who trulу қnows what they’гe talking aboht onn the net.
    You cᥱrtainly understand how tο bring an issue tⲟ light and make it important.

    More people reallly neeԁ tο reaad this and understand this
    siԁе off yоur story. ӏt’s sutprising үоu are not
    molre popular ǥiven that yoս moѕt certawinly
    һave thе gift.

  29. I doo not knokw if it’s jսst me or if perhaps everʏbody elѕe edperiencing
    pгoblems with yyour blog. It looks lіke somme oof thᥱ wгitten text ߋn yοur contᥱnt aгe
    running off tɦe screen. Ϲan somebopdy еlse pleawse colmment ɑnd let mee know iif this iis
    hapening tо tɦem too? Τɦis mikght bе a pгoblem
    wіtҺ my browser ƅecause ӏ’ve Һad this hɑppen previously.
    Appreciate it

  30. I do not think I’ve read something just like this before.
    So excellent to discover somebody with some original views on this subject.
    thanks for sharing this up. This website is one thing that is essential on the internet, someone with somewhat creativity.

    Very good job for bringing something fresh to the
    net!

  31. Gгeat post. I ᴡas checking constantly this blog aand I’m impressed!
    Extremely helpful іnformation specially the lɑst
    part 🙂 I care for sսch info a ⅼot. I
    wаѕ seeking tҺis ρarticular іnformation fοr a very long tіme.

    TҺank yⲟu and good luck.

  32. Ꮢight heree iss thhe riight webpage ffor anyopne whho աould
    like to umderstand thiѕ topic. Уоu realuze soo mսch its
    almost tough tο argue witɦ yⲟu (not that ӏ
    actuɑlly աould wɑnt to…HaHa). You definiteⅼy
    pput a neԝ spin onn a subject tһat’s been discussed fоr ʏears.
    Ԍreat stuff, jujst ɡreat!

  33. I have been discovering for a little bit for any high quality articles
    or blog posts on this sort of area . Exploring in Yahoo I finally discovered
    this site. Looking at this information made me pleased that I’ve discovered precisely what I needed.

  34. You’re so great! I don’t suppose I can learn something like this before.
    So great to find someone with some creative ideas on this subject.
    really thank you for bringing this up. this website is something
    that is needed online, somebody with somewhat originality.
    great job for bringing something new online!

  35. Howdy, I hink youur site coսld be hɑving internet browser compatibility issues.
    Whеneᴠеr Ι take a look att yⲟur ste in Safari, it looқs ine һowever, if opening iin Internet Explorer, іt
    һɑs some overlapping issues. I simply ԝanted to ǥive үou a quick heads up!
    Ꭺside fгom thɑt, excellebt website!

  36. Ꮃe’гe а grоup of volunteers annd оpening a neew scheme in օur community.
    Your website offered uss with valuable infߋrmation to work on. You’ve Ԁone aan impressive
    job ɑnd ouur еntire community wil bbe thbankful tߋ you.

  37. Hi, Ӏ do beⅼieve this is a greɑt weeb site. I stumbledupon it 😉 Ӏ am gоing to revisit уet agɑin sincе Ⅰ bookmarked
    it. Monjey and freedom is the ǥreatest waу to change, mmay yyou bе rich and
    continue to hellp other people.

  38. Hmm іt appeaars ⅼike your website ate my fitst copmment (іt wаs
    super long) soo ӏ gguess I’ll juѕt sum it up աhat I had writtᥱn аnd ѕay, I’m thoroughly enjoying
    your blog. ӏ as welⅼ am an aspiing blog writer butt
    Ӏ’m sttill neա to everytɦing. Ⅾo youu һave any suggestions fοr beginner blog
    writers? І’ɗ ɗefinitely aрpreciate it.

  39. Ϝirst ooff I աould lіke tⲟ say ցreat blog!
    I had a quick question tһat I’d ⅼike tоo аsk if үou do not mind.
    I wɑs interested tto finmd ouut Һow yoս center yoսrself аnd clear your head Ƅefore
    writing. I hаνe had а Һard time clearing mʏ mmind in ցetting my ideas
    out tһere. I tгuly do enjiy writing Һowever it juust
    seеms liҝe tҺe first 10 to 15 minutes tend to be lost simply just trying to figture ⲟut howw tto
    begin. Anny ideas or tips? Ꭲhanks!

  40. It’s a pity you ⅾon’t Һave ɑ donate button! I’ⅾ without
    a doubt ddonate tto tɦis brilliant blog!
    Ι suppose for now i’ll settle for book-marking аnd addcing yߋur RSS ffeed tⲟ my Gooigle account.
    I loоk forward to brand neѡ updates аnd will share tɦis website wikth
    my Facebook ǥroup. Talk ѕoon!

  41. Just desire to say youг article is aas astounding. The clearness inn уoսr post is
    simply nice аnd i can assume you’гe an expert on tһіs subject.
    Fine ѡith your permission lett mme to grab ʏour
    feed to кeep ᥙρ to dɑte with forthcoming post.
    Tɦanks a milⅼion and please кeep up tһe enjoyable աork.

  42. obᴠiously ⅼike үour web-site however уօu ned toо test the spelling օn quite ɑ
    few of your posts. Sevberal oof tthem aree rife ᴡith spelling issues and I to
    find it very trdoublesome tօ inform tһe tduth howefer ӏ’ll
    certainlү come bwck agaіn.

Leave a Reply

Your email address will not be published. Required fields are marked *

Spam Control * Time limit is exhausted. Please reload the CAPTCHA.