Friday, September 25, 2015

Introduction to ESP8266/EX development

In this post I'll present a step-by-step beginners guide for setting up a software development environment for ESP8266/EX.
Hardware issues are out of the scope for this post, and my assumption is that you have a working unit and a hardware utility you're familiar flashing it with.

The purpose of this post is to help you to skip the error mess resulted when trying to compile an open source project.

This guide will focus on a Linux setup, as it's pretty trivial for this kind of work.

The following components will be required:

  • Cross compiler - as ESP runs on a Xtensa architecture
  • SDK
  • Firmware sources
  • Flashing utilities
  • ESP unit

Begin with installing the dependencies packages:
sudo apt-get update
sudo apt-get install make unrar autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python python-serial sed git

Afterwards, clone the git esp-open-sdk:
cd /home/username/Documents
mkdir Espressif
cd Espressif
git clone --recursive https://github.com/pfalcon/esp-open-sdk.git

Building it might take a few minutes, and is done through:
cd esp-open-sdk
make STANDALONE=y

Note: when newer SDKs are released, updating the SDK is as simple as:
make clean
sudo git pull
sudo git submodule sync
sudo git submodule update

At this stage you have the requirements for building firmwares. This sample will show how to build the esphttpd project.
clone the httpd project:
cd ../
git clone --recursive https://github.com/izhak2/esphttpd
cd esphttpd

Edit the makefile according to the environment you just set up:
XTENSA_TOOLS_ROOT ?= /home/username/Documents/Espressif/esp-open-sdk/xtensa-lx106-elf/bin
SDK_EXTRA_INCLUDES ?= /home/username/Documents/Espressif/esp-open-sdk/sdk/include
SDK_EXTRA_LIBS ?= /home/username/Documents/Espressif/esp-open-sdk/sdk/lib
SDK_ROOT ?= /home/username/Documents/Espressif/esp-open-sdk/

Since during build xtensa-lx106-elf-nm is expected to be called as xt-nm, copy it:
cp /home/username/Documents/Espressif/esp-open-sdk/xtensa-lx106-elf/bin/xtensa-lx106-elf-nm /home/username/Documents/Espressif/esp-open-sdk/xtensa-lx106-elf/bin/xt-nm

Make the binaries available through the PATH environment variable:
PATH=/home/username/Documents/Espressif/esp-open-sdk/xtensa-lx106-elf/bin:$PATH

Finally, begin building. Building has two parts - there's the firmware and the mkespfsimage.
Both depend on the Heatshrink project, but with different settings.
Once completed building mkespfsimage, clean the Heathsrink build prior to building the firmware. Otherwise you might experience linking errors with some memory allocation functions.

First build mkespfsimage. It's used to generate the espfs (file-system over a single file, which is mapped as a whole to the flash).
cd mkespfsimage
make
and as explained, clean:
cd ../
make clean
Change heatshrink configuration to fit the firmware build; edit:
/home/username/Documents/esphttpd/lib/heatshrink/heatshrink_config.h
and perform the following changes:
"malloc" to "os_malloc"
"free" to "os_free"

Now, build the firmware using:
make rawflash ota=true
The make rawflash step will result in the firmware and espfs files available at /firmware, and a try to flash the files through serial.

You may also flash the files manually. For that procedure you may use (windows for now):
/utils/flash/ESP8266Flasher.exe
Additional files, except for firmware and espfs ,required for flashing will be found at:
/home/username/Documents/Espressif/esp-open-sdk/esp_iot_sdk_v1.3.0/bin
blank.bin
boot_v1.4(b1).bin

Map the offsets in ESP8266Flasher as following and flash to the appropriate COM port.
boot_v1.4(b1).bin at 0x00000
user1.new.bin at 0x01000
website.espfs at 0x41000
blank.bin at 0x7E000



Congratulations, now you have the firmware running on the ESP!

To view debug strings during runtime, if configured to be included, you may use PuTTY with 115200 speed serial configuration, and new line settings:
Terminal -> Implicit CR in every LF
to prevent new lines mess.

Finally, OTA flashing can be performed using the ota_flasher.py utility at /utils/flash/ota_flasher.py.
To perform OTA flashing you should have access to the chip of course, either if it's configured as a hotspot, or if connected to some other network component.
Have the ota_flasher.py and curl.exe as well as the firmware files and espfs at the same directory (verify the file names to match inside the script), and flash with the chip ip address.
On default, for example:
ota_flasher.py 192.168.4.1
When flashing firmware, wait between firmware and file-system flashing, since the chip performs a reset once firmware completed uploading.
Therefore, you may abort the script once asked for 'y' to flash the file-system, then re-execute the script skipping the firmware part, just to check if it recognizes the new boot partition.
Boot mode check at the prior to the flashing is done also for that - to make sure the chip is alive and responding, therefore ready for flashing.

Few more notes on OTA flashing -
Once flashed a firmware, you have to flash a file-system as well; although, you may flash only a file-system without replacing the firmware.

Credits:
Special thanks to my friend Boris.
Thanks to pfalcon who maintains the esp-open-sdk project which provides the relevant scripts for building a complete SDK.

7 comments:

  1. Hi
    I tried to build with open sdk 1.4.0 but got error
    section `.irom0.text' will not fit in region `irom0_0_seg'

    ReplyDelete
  2. Hi Dao,

    The error you mentioned is raised when the firmware size is too big for the specified flash size.
    Please consider the following steps:
    Check if your ESP has over 512KB; if so, change the ESP_FLASH_SIZE variable inside the Makefile to match (1024 for 1MB, etc.).
    Consider excluding debug strings to save space, by commenting out the relevant #define in config.h.
    Disable unnecessary (for your own use) functionality from the firmware. For now, this may be a bit annoying, but setting features' support from config.h is planned to be supported in the future.

    Good luck!

    ReplyDelete
  3. Hi, thank you for sharing this project. But after I download firmware to ESP8266, the webpage shows "Not Found." at http://192.168.1.10/index.tpl, I don't know why, SDK is v1.3.0, I will appreciate if if you can give me suggestions. Thanks.

    ReplyDelete
  4. Hi,

    Please verify that you changed the url in the firmware to serve requests for 192.168.1.10, because by default it's accessible by 192.168.4.1.
    However, if doesn't work, please contact me by email (izhak2 at gmail) and I'll try and help you.

    Good luck!

    ReplyDelete
  5. Hi,

    i'm trying to follow your instructions, but after cloning and building im missing the directory '/home/username/Documents/Espressif/esp-open-sdk/esp_iot_sdk_v1.3.0'.
    Where or how do i get it?

    Thanks

    ReplyDelete
  6. Hi again,

    i made some progress. When i'm connceting to the ESP8266 i get the response 'not found'. On the serial port i get following information:
    httpdParseHeader: URL is /index.tpl
    httpdParseHeader: URL index is 2
    espFsOpen: magic mismatch - file system image broken. found 0x88a0f530 instead
    /index.tpl not found. 404!
    httpdSendCb: connection 0x3fff2d98 is done and closing.
    There is no error when i build with : 'make website'

    Any idea why my image is broken and how i could fix it?

    Thanks miles

    ReplyDelete
    Replies
    1. Hi,

      Seems like you've not flashed the fs correctly, as the fw fails to access it properly. Verify addresses and flash size.
      If anything go wrong, remember that full flash dump is supported, where you can look for the signature found compared to the espfs magic, then test for the offset difference.

      Good luck!

      Delete