Idealvin Co Versions Save

Changes

• Fix initialization of static variables, see #345.
• Support non-ascii path in file APIs on Windows, see #347.
• Add fs::mv.
• Fix a bug in co::vector, see #349.
• Fix return value of _vm_reserve in memory allocator.
• Fix c_str() of fastring, fastream.

New features

• Benchmark framework.

#include "co/benchmark.h"
#include "co/mem.h"

BM_group(malloc) {
     void* p;

     BM_add(::malloc)(
         p = ::malloc(32);
     );
     BM_use(p);

     BM_add(co::alloc)(
         p = co::alloc(32);
     );
     BM_use(p);
}

int main(int argc, char** argv) {
     flag::parse(argc, argv);
     bm::run_benchmarks();
     return 0;
}

• gen supports generating code for converting JSON to struct, refer to the example test/j2s.

Improvement

• Coroutine scheduling optimization.
• Improve coroutine-related components. co::mutex, co::event, co::chan, etc. can be used in coroutines or non-coroutines, and POD types like std::string can be stored in co::chan.

Others

• Removed Random class, provided co::rand(), co::randstr(), and renamed the header file to co/rand.h`.
• Remove the global Thread and Mutex.
• APIs such as co::*scheduler* are renamed to co::*sched*.
• Other improvements, bug fixes, etc.

New features & improvement

• Improve cmake scripts and support Conan. Improve export of symbols in shared library. Thanks to SpaceIm.

• A better JSON library. Support precision control for float point numbers in Json.

  // {"a":23,"b":false,"s":"xx","v":[1,2,3],"o":{"xx":0}}
  Json x = {
      { "a", 23 },
      { "b", false },
      { "s", "xx" },
      { "v", {1,2,3} },
      { "o", {
          {"xx", 0}
      }},
  };
  
  // equal to x
  Json y = Json()
      .add_member("a", 23)
      .add_member("b", false)
      .add_member("s", "xx")
      .add_member("v", Json().push_back(1).push_back(2).push_back(3))
      .add_member("o", Json().add_member("xx", 0));
  
  x.get("a").as_int();       // 23
  x.get("s").as_string();    // "xx"
  x.get("v", 0).as_int();    // 1
  x.get("v", 2).as_int();    // 3
  x.get("o", "xx").as_int(); // 0
  

• Add TLOG to print logs which are grouped by topics.

  TLOG("rpc") << "hello " << 23;
  TLOG("xxx") << "hello " << 23;
  

• Improve exception handling in co/log on Windows. Thanks to 909254.

• Remove path from __FILE__ in co/log.

• Support alias for flag. Add api flag::alias().

  DEF_bool(debug, false, "");         // no alias
  DEF_bool(debug, false, "", d);      // d is an alias of debug
  DEF_bool(debug, false, "", d, dbg); // 2 aliases
  

• Ignore non-existing flags in config file.

• Add --version for co/flag.

• --help shows only user-defined flags in co/flag by default, and limit flag level to [0-9].

• Add api flag::set_value() for setting value of a flag.

• Add log::set_write_cb() to set a callback for writting logs.

• Add macros __arch64, __arch32, __fname__ and __fnlen__ in co/def.h.

• Add a fast memory allocator. Provide co::stl_allocator for STL.

• Provide containers based on co::stl_allocator. See co/stl.h for details.

• Improve atomic operations, support memory order. Add atomic_cas as an alias of atomic_compare_swap and add atomic_bool_cas.

• Add nanoid(), md5digest(), sha256() in co/hash.h.

• Support HTTP protocol for JSON-based RPC framework.

• Support graceful exit for tcp::Server, http::Server, rpc::Server.

• Add fs::dir for readding directory.

• Support + mode for both read and write for fs::file.

• Support precision control for float point numbers.

  fastream s;
  s << 3.14159; // "3.14159"
  
  s.clear();
  s.maxdp(3) << 3.14159; // "3.141"
  
  s.clear();
  s << co::maxdp(2) << 3.14159; // "3.14"
  
  LOG << co::maxdp(2) << 1.234e-7; // "1.23e-7"
  

• Improve hook on macos with fishhook. Thanks to shuai132.

Changed

• Remove flag a from co/unitest, run all tests by default.

• Remove log::init(), log::close(), log::set_single_write_cb().

• Remove co::init(), co::exit().

• Rename co::all_schedulers() to co::schedulers().

• Rename co::Table to co::table.

• Remove atomic_get, use atomic_load instead.

• Remove atomic_set, use atomic_store instead.

• Remove atomic_reset, use atomic_store(&x, 0) instead.

• Remove err::get(), err::set(), use co::error() instead.

Bugs fixed

• Fix hook for ioctl. Thanks to Itmit.

• Fix hook for accept & accept4.

• Fix a bug in co::Event.

New features

• Add str::cat(...), fastream::cat(...), fastring::cat(...)

fastring s("hello");
s.cat(' ', 23, "xx", false);          // s -> "hello 23xxfalse"
s = str::cat("hello", ' ', 23, true); // s -> "hello 23true"

• Support writing logs to customed destinations by setting a writing callback in co/log.

• Support daily rotation and log compression for co/log. Thanks to kuyoonjo. Compression is experimental at present.

• Add method reset() for fastring & fastream.

• Add method exit() for http::Server & rpc::Server.

• Add co::maybe.

• Add os::system().

• Add some god-oriented programming features in co/god.h.

Changed

• Improve exit() method for tcp::Server.

• Memory optimization for coroutines.

• Improve operator<< for fastream & fastring, optimization for string literal.

Bugs fixed

• Fix a bug in destructor of co::Event.

• Fix memory leak on windows.

• Fix a bug at exit in #189.

New features

• Support mingw on windows.

xmake f -p mingw
xmake -v

• Support shared library on windows.

xmake f -k shared
xmake -v

• Support syslog header for co/log.

xmake b log
xmake r log -syslog -cout

• Add god mode. When you feel uneasy about writing a function of several hundred lines, you may call the following api to increase your confidence:

#include "co/god.h"

god::bless_no_bugs();

Changed

• Add co/cout.h, make COUT and CLOG thread-safe.

• Add co::thread_id(), which may replace current_thread_id() in the future.

• co/log: add time in name of old log files.

• co/log: improve stack trace, use libbacktrace instead of fork()+backtrace().

• Add os::env(name, value) for setting value of environment variables.

• Always use / as the path separator, and convert \ to / in results of os::cwd(), os::homedir(), os::exepath().

• Improve the path library, support windows path like d:/xx.

• Improve stability of coroutine hook.

• tcp::Server::on_connection(tcp::Connection*) to tcp::Server::on_connection(tcp::Connection), no need to delete the connection object any more.

• Refactor http::Client, use curl easy handle only.

• Refactor http::Server, reduce memory copy to improve the performance.

Bugs fixed

• Fix RemoveVectoredExceptionHandler bug.

• Fix crash of http::Client in #168.

• Fix macro comment directive bug.

• Fix bug in select hook in #176.

• Fix a errno bug in co/flag, set errno to 0 before set_value.

• Fix errno bug in str::to_xxx().

• Fix hang-at-exit bug when using dll on windows.

• Fix iterator bug in std::multimap on windows.

• Fix: failed to hook fcntl with a different _FILE_OFFSET_BITS.

Documents

• English: github gitee
• 中文: github gitee

New features

• xrepo

xrepo install -f "openssl=true,libcurl=true" co

• vcpkg

vcpkg install co:x64-windows

# http & ssl support
vcpkg install co[libcurl,openssl]:x64-windows

• defer (similar to defer in golang)

#include "co/defer.h"
Timer t;
defer(LOG << "time elapse: " << t.us() << "us");

• channel for coroutine (similar to channel in golang)

#include "co/co.h"

DEF_main(argc, argv) {
    co::Chan<int> ch;
    go([ch]() {
        ch << 7;
    });

    int v = 0;
    ch >> v;
    LOG << "v: "<< v;

    return 0;
}

• waitgroup (similar to sync.WaitGroup in golang)

#include "co/co.h"

DEF_main(argc, argv) {
    FLG_cout = true;

    co::WaitGroup wg;
    wg.add(8);

    for (int i = 0; i <8; ++i) {
        go([wg]() {
            LOG << "co: "<< co::coroutine_id();
            wg.done();
        });
    }

    wg.wait();
    return 0;
}

• Coroutine hook for windows.

• Create coroutines in specified scheduler(thread).

auto s = co::next_scheduler();
s->go(f1);
s->go(f2);

• Create coroutine in all schedulers.

auto& s = co::all_schedulers();
for (size_t i = 0; i <s.size(); ++i) {
    s[i]->go(f);
}

• Add void flag::init(const fastring& path);

Changed

• Closure to co::Closure.

• Improve co::Event, can be used anywhere, and support copy constructor and capture by value in lambda.

• Improve co::Mutex, co::Pool, support copy constructor and capture by value in lambda.

• co::close() now can be called anywhere, not necessary to call it in the thread that performed the IO operations.

• Partial support for mingw. Coroutine and coroutine-based features do not work for mingw at present.

Bugs fixed

• fix bug in fs::file when read/write more than 4G bytes.

• fix connect timeout error for http::Client.

• fix link problem in #165.

Detailed reference documents

First of all, it is particularly emphasized that more detailed documents are provided this time:

• 中文 https://www.yuque.com/idealvin/co
• English https://www.yuque.com/idealvin/co_en

New features

SSL

Co 2.0 finally supports SSL. Users need to install openssl 1.1 or above. It has been tested on openssl and other SSL libraries have not been tested yet.

It is recommended to use xmake as the build tool, it will prompt the user whether to install openssl, libcurl, zlib and other third-party libraries. To enable the SSL feature, you need to predefine the CO_SSL macro, which is automatically defined by xmake when openssl is detected.

co/so/ssl.h provides a coroutine-based openssl interface, but users may not use it directly, because co has embedded the SSL feature into the TCP module, and users can use tcp::Server and tcp::Client instead.

Important improvements

Coroutine

• go
  go() supports any function or class method with 0 or 1 parameter, as well as function objects or pointers of type std::function<void()>.

  void f();
  void g(int);
  void h(int, int);
  struct T {
  	void f();
  	void g(int);
  };
  
  T o;
  std::function<void()> k(std::bind(h, 3, 7));
  
  go(f);
  go(g, 7);
  go(&T::f, &o);
  go(&T::g, &o, 3);
  go(k);
  go(&k); // The user must ensure that k is alive when the coroutine is running.
  

• Coroutine API

  • Adding the co::timeout() function, users can use it to determine whether the last I/O function like co::recv or co::send has timed out.
  • The co::coroutine_id() function returns a globally unique id. In version 1.x, coroutines in different scheduling threads may have the same id.

• co::Event
  The signaled state is added internally to solve the problem that the synchronization signal will be lost when there is no waiting coroutines.

• co::IoEvent
  In the 1.x version, it is only used internally, and this class is public in 2.0, so that users can coroutineize third-party network libraries by themselves.

  int recv(SSL* s, void* buf, int n, int ms) {
      CHECK(co::scheduler()) << "must be called in coroutine..";
      int r, e;
      int fd = SSL_get_fd(s);
      if (fd <0) return -1;
  
      do {
          ERR_clear_error();
          r = SSL_read(s, buf, n);
          if (r> 0) return r; // success
          if (r == 0) {
              DLOG << "SSL_read return 0, error: "<< SSL_get_error(s, 0);
              return 0;
          }
  
          e = SSL_get_error(s, r);
          if (e == SSL_ERROR_WANT_READ) {
              co::IoEvent ev(fd, co::EV_read);
              if (!ev.wait(ms)) return -1;
          } else if (e == SSL_ERROR_WANT_WRITE) {
              co::IoEvent ev(fd, co::EV_write);
              if (!ev.wait(ms)) return -1;
          } else {
              DLOG << "SSL_read return "<< r << ", error:" << e;
              return r;
          }
      } while (true);
  }
  

  
  The above is an example of coroutineizing the SSL_read() function in openssl. You only need to use a non-blocking socket. When openssl generates an SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE error, call the wait() method of co::IoEvent and wait for the corresponding I/O Event.
  
  

TCP

• Added the tcp::Connection class for the implementing TCP server. This class provides recv(), send() and other methods. Users can directly use this class to receive and send data without worrying about whether the underlying SSL is enabled.

• tcp::Server

  void on_connection(tcp::Connection* conn);
  
  tcp::Server s;
  s.on_connection(on_connection);
  s.start("0.0.0.0", 7788); // no ssl
  s.start("0.0.0.0", 7788, "privkey.pem", "certificate.pem"); // use ssl
  

  Users can specify the SSL private key and certificate file in the start() method to enable SSL.

• tcp::Client

  bool use_ssl = false;
  tcp::Client c("127.0.0.1", 7788, use_ssl);
  c.connect(1000);
  c.send(...);
  

  
  tcp::Client can enable SSL through the third parameter of the constructor.
  
  

HTTP

• http::Client
  In co 2.0, http::Client is implemented based on libcurl. To enable this feature, you must install libcurl and predefine the HAS_LIBCURL macro. Again, it is recommended to use xmake to build, it will automatically handle these third-party dependencies.

  http::Client c("https://github.com");
  http::Client c("http://127.0.0.1:7777");
  c.get("/");
  c.get("/idealvin/co");
  LOG << c.response_code();
  

• http::Server

  http::Server s
  s.on_req(...);
  s.start("0.0.0.0", 7777); // http
  s.start("0.0.0.0", 7777, "privkey.pem", "certificate.pem"); // https
  

RPC

In co 2.0, some new features have been added to the RPC framework:

• Support SSL.
• Support username and password authentication, and multiple usernames and passwords can be set for rpc::Server.
• Multiple services can be added into rpc::Server.
• The code generator can also generate code for RPC client.

JSON

In the 1.x version, the JSON library uses only one json::Value class to represent a JSON. The elements in the JSON object are also json::Value. When constructing a JSON object, you need to allocate memory for each element. When the JSON object is destructed, All internal elements must call the destructor of json::Value. Implementation based on this method will cause frequent memory allocation and release, which greatly affects program performance.

In co 2.0, the JSON object is built into a continuous memory. After the program runs stably, the parsing of JSON requires almost no memory allocation, which greatly improves the parsing speed.

In addition, co 2.0 uses the Json class to represent a JSON, and the json::Value class to represent the elements in the JSON. json::Value is just a trivial class, only including a index position in the JSON memory block. When a JSON is destructed, the destructor of the Json class is called only once, and the destructor of json::Value will never be called.

Others

• Fix the nested log bug in co/log.
• Fix some bugs caused by dependence of global static variables.
• Add milliseconds in log time of co/log.
• The TaskSched class is renamed to Tasked.
• co/time.h. Add epoch::ms() and epoch::us() to obtain the time since the EPOCH.
• co/os.h Add os::signal() method to set the signal handler function.
• Added safe_clear() method in fastring.
• Added safe_clear() method in Json.

• fix #118
• update vs project

fix #99