Monday, September 4, 2017

Arduino based simple weather station with web interface

... an IoT idea,

 To build a device and a web interface to sense, store and display an environmental data ...

 Here is the idea …
  • Design a device to capture some environmental data,
    • Sensor to report humidity and temperature
    • Sensor to report ambient light
    • Catch them every 2 minutes approx., (or any interval)
    • Send the captured data to cloud for recording and reporting
  • Design a web service,
    • A simple web API to receive and store data
    • A simple process to plot captured data as a graph 
    • A web page to display the graph

 What you need,
  • Sensors
  • Ambient light level sensor
  • Internet connectivity
  • Micro-controller to connect all these
    • Arduino UNO – what I used here
      • https://store.arduino.cc/usa/arduino-uno-rev3
      • … and some jumper cables to connect 
      • … and an Arduino C program to capture and send data
  • A cloud based web server
    • Google compute cloud server is used for this
    • A web app to capture data – Python Flask app
      • A POST request is the ideal method (... could not get it to work :( )
      • A GET method worked, thus implemented it …
    • A web app to represent captured data – same app above …

 Idea envisioned on a Frtizing design,

Wire the Arduino and related circuit on a breadboard like this. Arduino board's own 3.3v is used for BH1750 light sensor. We need a high amperage 3.3v supply to ESP 8266 WiFi module. That is why KIA78R33 a 3.3v voltage regulator is used to power the ESP. BHT11 sensor can take 5v directly from Arduino.

 Idea materialized,



Real time data,


Code … ‘the fun begins here ...',

Code and explanation,

First the Arduino C code,

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
/*
   Environmental Data Sensing and Storing
   mscs crysanthus@gmail.com
*/
#include <SoftwareSerial.h>
#include <Adafruit_Sensor.h>
#include <DHT.h>
#include <DHT_U.h>
#include <Wire.h>
#include <BH1750FVI.h>

SoftwareSerial homeStatESP(6, 7); // RX, TX // ESP Module

DHT_Unified dhtSensor(2, DHT11); /// DHT sensor

BH1750FVI bhLightSensor; // BH1750 Ambient Light Sensor
/*
  sensor connected
  VCC >>> 3.3V
  SDA >>> A4
  SCL >>> A5
  addr >> A3
  Gnd >>>Gnd
*/

uint32_t delayMS;

String asSensorData[3];

// SendCommand prototype
boolean SendCommand(String sATCmd, String sResponse, boolean bCRLF = true);

void setup()
{
  Serial.begin(9600);

  homeStatESP.begin(115200);

  /* --- SENSORS
  */
  // dht sensor
  dhtSensor.begin();

  sensor_t sensor;
  delayMS = sensor.min_delay / 1000;

  // light sensor
  bhLightSensor.begin();
  bhLightSensor.SetAddress(Device_Address_H);//Address 0x5C
  bhLightSensor.SetMode(Continuous_H_resolution_Mode);

  // coonect to ap
  while (!JoinAP());

}

void loop() {

  // Get Temp
  sensors_event_t event;
  dhtSensor.temperature().getEvent(&event);
  if (isnan(event.temperature)) {
    Serial.println("Error reading temperature!");
  }
  else {
    asSensorData[0] = (String)int(event.temperature);
  }

  // Get Humd
  dhtSensor.humidity().getEvent(&event);
  if (isnan(event.relative_humidity)) {
    Serial.println("Error reading humidity!");
  }
  else {
    asSensorData[1] = (String)int(event.relative_humidity);
  }

  // Get Lux / Lumn
  uint16_t lux = bhLightSensor.GetLightIntensity();
  asSensorData[2] = (String)int(lux);

  Serial.println(asSensorData[0] + "*C - " + asSensorData[1] + "% - " + asSensorData[2] + "lux");

  PostData(asSensorData);

  // Delay between measurements.
  delay(60000);
}

/*
   ESP Init
*/
void StartESP() {
  Serial.println("ESP init!");
  SendCommand("AT+RST", "OK");
}

/*
   AT Commander
*/
boolean SendCommand(String sATCmd, String sResponse, boolean bCRLF = true) {
  Serial.println("CMD rcvd!");
  Serial.println(sATCmd);
  bCRLF ? homeStatESP.println(sATCmd) : homeStatESP.print(sATCmd);
  delay(8000);
  return responseFind(sResponse); // find given response
}

/*
   AT Response find
*/
boolean responseFind(String sKey) {

  long lTO = millis() + 5000; // Time Out

  char cKey[sKey.length()];
  sKey.toCharArray(cKey, sKey.length());

  while (millis() < lTO) {

    if (homeStatESP.available()) {

      if (homeStatESP.find(cKey)) {
        Serial.println(sKey);
        return true;
      }
    }
  }

  return false; // timeout
}

boolean JoinAP() {
  Serial.println("Joining AP ...");

  String sAP = "AP NAME";
  String sPP = "AP PASSWORD";

  StartESP();

  if (SendCommand("AT+CWJAP=\"" + sAP + "\",\"" + sPP + "\"", "OK")) {
    Serial.println("Joined AP!");
    return true;
  }
  else {
    Serial.println("Failed joining AP ...");
    return false;
  }

}

/*
   Post to Web API
   a Python Flask API waiting for data
*/
void PostData(String saData[]) {

  String sServer = "WEB SERVER IP OR NAME"; // Without http part
  String sPort = "5000";
  String sURI = "/api/addstatsg?";

  if (SendCommand("AT+CIPSTART=\"TCP\",\"" + sServer + "\"," + sPort, "OK")) {
    Serial.println("TCP connection ready!");

    /**
       HTTP GET Sample
       GET /api/addstatsg?cert=xyz&amp;temp=34&amp;humd=45&amp;lumn=55 HTTP/1.1
       Host: 192.168.1.5:5000
    */
    String sPostRequest =  String("GET " + sURI);
            sPostRequest += String("cert=xyz&");
            sPostRequest += String("temp=" + saData[0] + "&");
            sPostRequest += String("humd=" + saData[1] + "&");
            sPostRequest += String("lumn=" + saData[2] + " HTTP/1.1\r\n");
            sPostRequest += String("Host: " + sServer + ":" + sPort + "\r\n\r\n"); // ends with 2 x CRLF - Important -
    /*
       This is by AT design
    */
    SendCommand("AT+CIPSEND=", "", false);
    if (SendCommand(String(sPostRequest.length()), ">")) {
      Serial.println("TCP ready to receive data ...");

      if (SendCommand(sPostRequest, "SEND OK")) {
        Serial.println("Data packets sent ok!");

        while (homeStatESP.available()) {
          Serial.println(homeStatESP.readString());
        }
        // close the connection
        CloseTCPConnection();
      }
      else {
        CloseTCPConnection();
      }
    }
    else {
      CloseTCPConnection();
    }
  } else {
    CloseTCPConnection();
  }
}

/*
   Close TCP connection
*/
void CloseTCPConnection() {

  while (!SendCommand("AT+CIPCLOSE", "OK")) {
    Serial.print(".");
  }
  Serial.println("TCP connection closed!");
}

/*
   EOF
*/

  1. Upload this code to Arduino UNO
  2. Replace the following with your own settings - your own WiFi SSID / Access Point name and password
  3. Web server IP or name without the http:// part. Can change the port but the same port number must be in the python code too in line 162 

  String sAP = "AP NAME";
  String sPP = "AP PASSWORD";

  String sServer = "WEB SERVER IP OR NAME"; // Without http part
  String sPort = "5000";
  String sURI = "/api/addstatsg?";

Once successful, remove all the Serial.println statements to save some space in Arduino.

And the python code,

... this code need three major libs from python - matplotlib, flask, gevent

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
#!flask/bin/python
#
# Environmental Data Sensing and Storing 
# mscs crysanthus@gmail.com
#
import datetime

from flask import Flask, abort, request, make_response, jsonify, render_template, send_from_directory

# db
import sqlite3

# graph
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

 
# WSGI Server
from gevent.wsgi import WSGIServer

# create db and table in init case
try:
 conn = sqlite3.connect('homestats.db')
 cursr = conn.cursor()
 cursr.execute('''CREATE TABLE IF NOT EXISTS weather_data
  (iid integer primary key, date text, temp text, humd text, lumn text)''') 
 conn.commit()

except:
 abort(500)

app = Flask(__name__, static_url_path='')

# serve images directly, no process
@app.route('/favicon.ico')
def send_favicon():
    return send_from_directory('templates/images', 'favicon.ico')

# serve images directly, no process
@app.route('/images/<path:path>')
def send_images(path):
    return send_from_directory('templates/images', path)
    
# serve graph directly, no process
@app.route('/graphs/<path:path>')
def send_graphs(path):
    return send_from_directory('templates/graphs', path)

# save data
# save GET data -- working part --
@app.route('/api/addstatsg', methods=['GET'])
def add_stats_get():

 try:
  # check to see if API token is correct -- TODO --
  if not request.args['cert']:
   abort(417)
 
  if request.args['cert'] != 'xyz':
    abort(417)
  # check to see if API token is correct -- TODO --
  
  # chk to see if the field have correct values
  if not request.args['temp'] or not request.args['temp'].isnumeric():
   abort(404)

  if not request.args['humd'] or not request.args['humd'].isnumeric():
   abort(404)

  if not request.args['lumn'] or not request.args['lumn'].isnumeric():
   abort(404)
 
  stats = {'date': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
   'temp': request.args['temp'],
   'humd': request.args['humd'],
   'lumn': request.args['lumn']
   }

 except:
  abort(501)
 
 try:
  conn = sqlite3.connect('homestats.db')
  cursr = conn.cursor()

  cursr.execute('''INSERT INTO weather_data 
   VALUES(NULL, :date, :temp, :humd, :lumn)''', stats)
  conn.commit()
  
 except:
  abort(500)

 return jsonify({'homestats':'data saved'}), 201


# create graph and serve stat html
@app.route('/')
def index():

 # do the graph
 plt.ylabel('Units')
 plt.xlabel('Date/Time')
 
 # get data
 conn = sqlite3.connect('homestats.db')
 cursr = conn.cursor()
 cursr.execute('''SELECT date, temp, humd, lumn FROM weather_data;''')
 xv = [] # x axis
 yv1 = [] # y axis - tempreture in red
 yv2 = [] # y axis - humidity in green
 yv3 = [] # y axis - ambient light in yellow
 
  # add data to graph
 for row in cursr:
  xv.append(datetime.datetime.strptime(row[0],"%Y-%m-%d %H:%M:%S"))
  yv1.append(myFloat(row[1]))
  yv2.append(myFloat(row[2]))
  yv3.append(myFloat(row[3]))

 # legends
 plt.plot(xv, yv1, '-r', label='Tempreture', linewidth=0.50)
 plt.plot(xv, yv2, '-g', label='Humidity', linewidth=0.75)
 plt.plot(xv, yv3, '-y', label='Light', linewidth=0.80)
 # beautify the x-labels
 plt.gcf().autofmt_xdate()
 plt.legend()
 
 # save graph as a .png file to serve via html
 plt.savefig('templates/graphs/statistic.png')
 plt.close()
 
 # -- send html out with graph --
 return render_template('stats.html')

# http errors
@app.errorhandler(404)
def not_found(error):
    return make_response(jsonify({'error': 'Not found'}), 404)

@app.errorhandler(500)
def not_found(error):
    return make_response(jsonify({'error': 'Server Internal error'}), 500)

@app.errorhandler(501)
def not_found(error):
    return make_response(jsonify({'error': 'Not Implemented'}), 501)

@app.errorhandler(417)
def not_found(error):
    return make_response(jsonify({'error': 'Token error'}), 417)

# chk to see if a string is float-able
def myFloat(s):
    try:
        return float(s)
    except ValueError:
        return 0.0

if __name__ == '__main__':
    http_server = WSGIServer(('0.0.0.0', 5000), app)
    http_server.serve_forever()

Hosting,
I set up a tiny Google cloud server with Ubuntu Linux 16.04 server to host this app. Need to do the following to add python dependencies. Need to have shell access to the server to do the following,
  1. Update and upgrade Ubuntu and install GCC compiler
    2. 1
2
3
    sudo apt update
sudo apt upgrade
sudo apt install gcc
  2. Install python tools
    3. 1
2
3
    sudo apt install python-dev
sudo apt install python-virtualenv 
sudo apt install python-pip
  3. Create app folders
    4. 1
2
3
    mkdir homestat
cd homestat
mkdir -p templates/{images,graphs}
  4. Create python flask environment
  5. Get inside flask environment
  6. Add dependencies to run web app
    7. 1
2
3
4
5
    virtualenv flask
source flask/bin/activate
pip install flask
pip install matplotlib
pip install gevent
  7. Create a HTML file called stats.html in homestats/flask/templates/ folder and copy the content below
    8. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
 <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
 <title>Home Stats</title>
 <style type="text/css">
  @page { margin: 0.79in }
  p { margin-bottom: 0.1in; line-height: 120% }
  img { display: block; margin: 0 auto;}
 </style>
</head>
<body lang="en-US" dir="ltr">
 <p align="center" style="margin-bottom: 0.25in; line-height: 100%">Home Stats</p>
 <img src="images/building-statistics.jpg" alt="building-statistics"/>
 <hr/>
 <img id="graph" src="graphs/statistic.png" alt="home-statistics"/>
 <script language="javascript" type="text/javascript">
      var d = new Date(); 
      document.getElementById("graph").src="graphs/statistic.png?ver="+d.getTime();
 </script>
</body>
</html>
  8. Upload a small graphic logo in homestats/flask/templates/images/ folder to be displayed on your web page. Name it building-statistics.jpg
  9. Upload a small favicon.ico in homestats/flask/templates/images/ folder. Not a must but a nice to have.
  10. Copy and paste the above python source code to a file named app.py and save.
  11. This web app is setup to listen to TCP port 5000. In Google cloud firewall rule is required to allow web traffic to port 5000. Goto VPC nework - Firewall rules to setup that.
  12. Run web app
    13. 1
    python ./app.py
  13. If everything go well, you must see the HTTP GET calls displayed as follows - which means Arduino is sending data correctly to your web app
  14. And now you can get the python app to create the statistic graph by calling http:<your web server IP>:5000/
  15. tips and tricks - I ran this python app inside a screen tool and close the terminal. It runs as a temporary daemon. Don't CTRL + C and exit. It will terminate the server.
  16. Let me know your success story
  17. The code and concept must be shared and spread!


Posted by at
Labels: , , , , , , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)