Category: Blog

  • CryptoMarket

    CryptoMarket

    CryptoMarket Logo

    CryptoMarket is a prototype. It is a Marketplace that runs over the Etherem Blockchain. It is able to manage products (and orders, too).

    Since it runs over the blockchain it doesn’t need to have any server.

    How use it

    You can run it by yourself or use my hosted files here.
    In order to run this files you have to use something that inject Web3 into the webpage, the simplest way is to use Chrome with MetaMask Extension installed.

    Remember, if you would like to use your machine to use it, you have to run a webserver (and not file://), otherwise MetaMask cannot inject the code due security restrictions.

    Contract

    The contract is developed using Solidity. It is actually deployed over Rinbeky Ethereum Test Net. You can you this code to understand how Solidity works. In order to deploy your own contract, you can refers to this page.

    API

    getOwnOrders() returns(uint[])

    Returns the ids of the orders that refers to product sold by the msg.sender.

    getProductIdsBySeller(address _seller) returns(uint[])

    Returns the ids of product sold by specific address.

    buyProduct(uint _id, uint32 _quantity, string _shippingAddress) payable

    Is the only payable function of the contract. If the msg.value is equal to the price of the product with id equals to _id, then decrease the product avaiable quantity and create an order.

    orders

    This is the dynamic arrays that stores all the orders created. The Order has the following structure

    struct Order {
            string shippingAddress;
            uint32 quantity;
            uint productId;
        }
    addProduct(string _n, string _d, uint _p, uint _q)

    This function add a product to products dynamic array. Set the linked proprieties stored into the following self-described maps:

    	mapping(uint => address) public productToSeller;
        mapping(address => uint32) public sellerProductCounter;

    At the end generated the following event NewProductCreated(address seller, uint productId);.

    removeProduct(uint _productId)

    This function delete the product from the products array. It can be executed only by the product’s seller.

    What’s next

    The orders are not yet implemented.

    Visit original content creator repository
    https://github.com/alessandro308/CryptoMarket

  • csv2sql

    CSV2SQL, An util to convert csv data to SQL scripts and load to MySQL.

    How it works

    Pipeline:

    [Read the xlsx file] -> [Convert to SQL scripts] -> [Execute SQL scripts]

    How to use

    1. Prepare the xlsx file with your data

    Assume the file is dropped into test/import.xlsx, you can download it from here

    1. Config the csv2sql.yml
    modules:
    - name: pipeline
      enabled: true
      runners:
        - name: process_excel
          enabled: true
          input_queue: primary
          max_go_routine: 1
          threshold_in_ms: 0
          timeout_in_ms: 5000
          default_config:
            start:
              joint: read_csv
              enabled: true
              parameters:
                file_name: "test/import.xlsx"
            process:
            - joint: convert_sql
              enabled: true
              parameters:
                sheet_name: fish_information
                data_start_from_index: 3
                column_name:
                - id
                - outer_code
                - common_name
                - scientific_name
                - english_name
                - chinese_name
                - region_name
                - aquatic_category_id
                - category_name
                - is_homemade
                - aquatic_region_id
                - inner_code
                - produce_pattern
                - feed_pattern
                - catch_pattern
                row_format:
                - 'INSERT INTO `aquatic_base_info` (`id`, `outer_code`, `common_name`, `scientific_name`, `english_name`, `chinese_name`, `region_name`, `aquatic_category_id`)'
                - 'VALUES (<{id: }>, <{outer_code: }>, <{common_name: }>, <{scientific_name: }>, <{english_name: }>, <{chinese_name: }>, <{region_name: }>, <{aquatic_category_id: }>);'
                - 'INSERT INTO `aquatic_source` (`inner_code`, `aquatic_base_info_id`, `is_homemade`, `aquatic_region_id`, `produce_pattern`, `feed_pattern`, `catch_pattern`) '
                - 'VALUES (<{inner_code: }>, <{id: }>, <{is_homemade: }>, <{aquatic_region_id: }>, <{produce_pattern: }>, <{feed_pattern: }>, <{catch_pattern: }>);'
    
            - joint: convert_sql
              enabled: false
              parameters:
                sheet_name: aquatic_region
                data_start_from_index: 1
                column_name:
                - id
                - code
                - name
                - is_homemade
                row_format:
                - 'INSERT INTO `aquatic_region` (`id`, `code`, `name`, `is_homemade`) VALUES (<{id: }>, <{code: }>, <{name: }>, <{is_homemade: }>);'
    
            - joint: import_sql
              enabled: true
              parameters:
                mysql_conn: root:password@tcp(localhost:3306)/ifish?charset=utf8
                rollback_enabled: true
    
            - joint: logging
              enabled: true
    
            error:
              joint: on_error
              enabled: true
    

    Note, there are more than one joint config with name: convert_sql, which means you can import multi data sheet with different config, in this example the second convert_sql joint has been disabled, you can enable it if you wish, and also you can add more convert_sql joints.

    And also as you can see, in the first convert_sql joint, with the data sheet fish_information, the config row_format is a array, and have more than one SQL template, separated with ; , which means you can generate multi SQL from one single data sheet, map one data row to multi mysql data records, and then we can inset the data into different mysql tables.

    The config row_format is a SQL template, and the config column_name is how your data sheet will be used in your template, like this template variable <{is_homemade: }>, it will looking for the data from column is_homemade which we have already configured in section column_name, you can get the SQL template by using MySQLWorkBench quickly([select db]->[select table]->[Copy to Clipboard]->[Insert Statement])

    Note, once you change the column in data sheet, you must keep the column_name updated.

    Note, please also change MySQL connection in joint mysql_conn config, ie: your_mysql_user:you_password@tcp(your_mysql_host:3306)/your_mysql_db?charset=utf8

    And, the example is jus a example, you can use it to map any data sheet to any mysql table, update the config and import the data.

    1. Start to import the data
    ➜  csv2sql git:(master) ✗ ./bin/csv2sql
      _____  _______      _____   _____  ____  _
     / ____|/ ____\ \    / /__ \ / ____|/ __ \| |
    | |    | (___  \ \  / /   ) | (___ | |  | | |
    | |     \___ \  \ \/ /   / / \___ \| |  | | |
    | |____ ____) |  \  /   / /_ ____) | |__| | |____
     \_____|_____/    \/   |____|_____/ \___\_\______|
    [CSV2SQL] An util to convert csv data to SQL scripts.
    0.1.0_SNAPSHOT,  67026ee, Sat May 5 13:23:42 2018 +0800, medcl, support import multi datasheet
    
    [05-05 15:29:38] [INF] [instance.go:23] workspace: data/APP/nodes/0
    [05-05 15:29:38] [INF] [pipeline.go:67] pipeline: process_excel started with 1 instances
    [05-05 15:29:38] [INF] [api.go:147] api server listen at: http://0.0.0.0:2900
    [05-05 15:29:38] [INF] [ui.go:149] http server listen at: http://127.0.0.1:9001
    [05-05 15:29:38] [INF] [import_sql.go:87] sql execute success, 1 rows affected, lastInsertID: 1
    [05-05 15:29:38] [INF] [import_sql.go:87] sql execute success, 1 rows affected, lastInsertID: 1
    ^C
    [CSV2SQL] got signal:interrupt, start shutting down
                             |    |
       _` |   _ \   _ \   _` |     _ \  |  |   -_)
     \__, | \___/ \___/ \__,_|   _.__/ \_, | \___|
     ____/                             ___/
    [CSV2SQL] 0.1.0_SNAPSHOT, uptime:5.773621s
    
    
    1. Now, check out the database, you will see 2 new records in two different table

    License

    Released under the Apache License, Version 2.0 .

    Powered by https://github.com/infinitbyte/framework

    Visit original content creator repository https://github.com/medcl/csv2sql
  • csv2sql

    CSV2SQL, An util to convert csv data to SQL scripts and load to MySQL.

    How it works

    Pipeline:

    [Read the xlsx file] -> [Convert to SQL scripts] -> [Execute SQL scripts]

    How to use

    1. Prepare the xlsx file with your data

    Assume the file is dropped into test/import.xlsx, you can download it from here

    1. Config the csv2sql.yml

    modules:
    - name: pipeline
      enabled: true
      runners:
        - name: process_excel
          enabled: true
          input_queue: primary
          max_go_routine: 1
          threshold_in_ms: 0
          timeout_in_ms: 5000
          default_config:
            start:
              joint: read_csv
              enabled: true
              parameters:
                file_name: "test/import.xlsx"
            process:
            - joint: convert_sql
              enabled: true
              parameters:
                sheet_name: fish_information
                data_start_from_index: 3
                column_name:
                - id
                - outer_code
                - common_name
                - scientific_name
                - english_name
                - chinese_name
                - region_name
                - aquatic_category_id
                - category_name
                - is_homemade
                - aquatic_region_id
                - inner_code
                - produce_pattern
                - feed_pattern
                - catch_pattern
                row_format:
                - 'INSERT INTO `aquatic_base_info` (`id`, `outer_code`, `common_name`, `scientific_name`, `english_name`, `chinese_name`, `region_name`, `aquatic_category_id`)'
                - 'VALUES (<{id: }>, <{outer_code: }>, <{common_name: }>, <{scientific_name: }>, <{english_name: }>, <{chinese_name: }>, <{region_name: }>, <{aquatic_category_id: }>);'
                - 'INSERT INTO `aquatic_source` (`inner_code`, `aquatic_base_info_id`, `is_homemade`, `aquatic_region_id`, `produce_pattern`, `feed_pattern`, `catch_pattern`) '
                - 'VALUES (<{inner_code: }>, <{id: }>, <{is_homemade: }>, <{aquatic_region_id: }>, <{produce_pattern: }>, <{feed_pattern: }>, <{catch_pattern: }>);'
    
            - joint: convert_sql
              enabled: false
              parameters:
                sheet_name: aquatic_region
                data_start_from_index: 1
                column_name:
                - id
                - code
                - name
                - is_homemade
                row_format:
                - 'INSERT INTO `aquatic_region` (`id`, `code`, `name`, `is_homemade`) VALUES (<{id: }>, <{code: }>, <{name: }>, <{is_homemade: }>);'
    
            - joint: import_sql
              enabled: true
              parameters:
                mysql_conn: root:password@tcp(localhost:3306)/ifish?charset=utf8
                rollback_enabled: true
    
            - joint: logging
              enabled: true
    
            error:
              joint: on_error
              enabled: true
    

    Note, there are more than one joint config with name: convert_sql, which means you can import multi data sheet with different config,
    in this example the second convert_sql joint has been disabled, you can enable it if you wish, and also you can add more convert_sql joints.

    And also as you can see, in the first convert_sql joint, with the data sheet fish_information, the config row_format is a array,
    and have more than one SQL template, separated with ; , which means you can generate multi SQL from one single data sheet,
    map one data row to multi mysql data records, and then we can inset the data into different mysql tables.

    The config row_format is a SQL template, and the config column_name is how your data sheet will be used in your template,
    like this template variable <{is_homemade: }>, it will looking for the data from column is_homemade which we have already configured in section column_name,
    you can get the SQL template by using MySQLWorkBench quickly([select db]->[select table]->[Copy to Clipboard]->[Insert Statement])

    Note, once you change the column in data sheet, you must keep the column_name updated.

    Note, please also change MySQL connection in joint mysql_conn config, ie: your_mysql_user:you_password@tcp(your_mysql_host:3306)/your_mysql_db?charset=utf8

    And, the example is jus a example, you can use it to map any data sheet to any mysql table, update the config and import the data.

    1. Start to import the data

    ➜  csv2sql git:(master) ✗ ./bin/csv2sql
      _____  _______      _____   _____  ____  _
     / ____|/ ____\ \    / /__ \ / ____|/ __ \| |
    | |    | (___  \ \  / /   ) | (___ | |  | | |
    | |     \___ \  \ \/ /   / / \___ \| |  | | |
    | |____ ____) |  \  /   / /_ ____) | |__| | |____
     \_____|_____/    \/   |____|_____/ \___\_\______|
    [CSV2SQL] An util to convert csv data to SQL scripts.
    0.1.0_SNAPSHOT,  67026ee, Sat May 5 13:23:42 2018 +0800, medcl, support import multi datasheet
    
    [05-05 15:29:38] [INF] [instance.go:23] workspace: data/APP/nodes/0
    [05-05 15:29:38] [INF] [pipeline.go:67] pipeline: process_excel started with 1 instances
    [05-05 15:29:38] [INF] [api.go:147] api server listen at: http://0.0.0.0:2900
    [05-05 15:29:38] [INF] [ui.go:149] http server listen at: http://127.0.0.1:9001
    [05-05 15:29:38] [INF] [import_sql.go:87] sql execute success, 1 rows affected, lastInsertID: 1
    [05-05 15:29:38] [INF] [import_sql.go:87] sql execute success, 1 rows affected, lastInsertID: 1
    ^C
    [CSV2SQL] got signal:interrupt, start shutting down
                             |    |
       _` |   _ \   _ \   _` |     _ \  |  |   -_)
     \__, | \___/ \___/ \__,_|   _.__/ \_, | \___|
     ____/                             ___/
    [CSV2SQL] 0.1.0_SNAPSHOT, uptime:5.773621s
    
    
    1. Now, check out the database, you will see 2 new records in two different table

    License

    Released under the Apache License, Version 2.0 .

    Powered by https://github.com/infinitbyte/framework

    Visit original content creator repository
    https://github.com/medcl/csv2sql

  • productDisplay

    Getting Started with Create React App

    This project was bootstrapped with Create React App.

    Available Scripts

    In the project directory, you can run:

    npm start

    Runs the app in the development mode.
    Open http://localhost:3000 to view it in your browser.

    The page will reload when you make changes.
    You may also see any lint errors in the console.

    npm test

    Launches the test runner in the interactive watch mode.
    See the section about running tests for more information.

    npm run build

    Builds the app for production to the build folder.
    It correctly bundles React in production mode and optimizes the build for the best performance.

    The build is minified and the filenames include the hashes.
    Your app is ready to be deployed!

    See the section about deployment for more information.

    npm run eject

    Note: this is a one-way operation. Once you eject, you can’t go back!

    If you aren’t satisfied with the build tool and configuration choices, you can eject at any time. This command will remove the single build dependency from your project.

    Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except eject will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own.

    You don’t have to ever use eject. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it.

    Learn More

    You can learn more in the Create React App documentation.

    To learn React, check out the React documentation.

    Code Splitting

    This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting

    Analyzing the Bundle Size

    This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size

    Making a Progressive Web App

    This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app

    Advanced Configuration

    This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration

    Deployment

    This section has moved here: https://facebook.github.io/create-react-app/docs/deployment

    npm run build fails to minify

    This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify

    Visit original content creator repository
    https://github.com/musmax/productDisplay

  • package

    Marrow Package

    © 2014-2023 Alice Bevan-McGregor and contributors.
    https://github.com/marrow/package
    Latest Version Release Build Status Release Test Coverage Github Issues

    1. What is Marrow Package?

    This package is a combination of utilities for handling object lookup, resolving object names, and managing simple to complex plugin architectures. Notably it includes a dependency graph system for extensions and helper for looking up qualified object names.

    This library is fully unit tested where possible.

    2. Installation

    Installing marrow.package is easy, just execute the following in a terminal:

    pip install marrow.package
    

    Note: We strongly recommend always using a container, virtualization, or sandboxing environment of some kind when developing using Python; installing things system-wide is yucky (for a variety of reasons) nine times out of ten. We prefer light-weight virtualenv, others prefer solutions as robust as Vagrant.

    If you add marrow.package to the install_requires argument of the call to setup() in your application’s setup.py file, Marrow Package will be automatically installed and made available when your own application or library is installed. We recommend using “less than” version numbers to ensure there are no unintentional side-effects when updating. Use marrow.package<2.2 to get all bugfixes for the current release, and marrow.package<3.0 to get bugfixes and feature updates while ensuring that large breaking changes are not installed.

    2.1. Development Version

    Development Build Status Development Test Coverage

    Development takes place on GitHub in the marrow.package project. Issue tracking, documentation, and downloads are provided there.

    Installing the current development version requires Git, a distributed source code management system. If you have Git you can run the following to download and link the development version into your Python runtime:

    git clone https://github.com/marrow/package.git
    (cd package; python setup.py develop)
    

    You can then upgrade to the latest version at any time:

    (cd package; git pull; python setup.py develop)
    

    If you would like to make changes and contribute them back to the project, fork the GitHub project, make your changes, and submit a pull request. This process is beyond the scope of this documentation; for more information see GitHub’s documentation.

    3. Getting Object References

    Object references describe the module and attribute path needed to resolve the object. For example, foo:bar is a reference that describes importing “foo” prior to retrieving an object named “bar” from the module. On Python 3.3+ a useful shortcut is provided, __qualname__ which speeds up this lookup.

    For example, let’s define a class and get a reference to it:

    from marrow.package.canonical import name
    
    class Example(object):
        pass
    
    assert name(Example) == '__main__:Example'
    

    You can, depending on platform, retrieve a reference to any of the following types of objects:

    • Module level:
      • class
      • class instance
      • class method
      • class staticmethod
      • function
      • instance classmethod
      • instance method
      • instance staticmethod
    • nested classes and methods
    • closures

    3.1. Resolving Plugin References

    The load utility can optionally be provided a plugin namespace to search. If the target object is found within the namespace, the name of the plugin entry will be returned. By default, if a named plugin can not be found, a LookupError will be raised. If a direct reference is acceptable, the boolean direct argument (third positional) can be made truthy to permit direct references.

    from marrow.package.canonical import name

    assert name(name, ‘marrow.package.sample’) == ‘name’

    4. Resolving Object References

    Two utilities are provided which allow you resolve string path references to objects. The first is quite simple:

    from marrow.package.loader import traverse
    
    assert traverse({'foo': {'bar': 27}}, 'foo.bar') == 27
    

    This will search the dictionary described for a “foo” element, then “bar” element.

    The traverse function takes some additional optional arguments. If executable is True any executable function encountered will be executed without arguments. Traversal will continue on the result of that call. You can change the separator as desired, i.e. to a “https://github.com/” using the separator argument.

    By default attributes (but not array elements) prefixed with an underscore are taboo. They will not resolve, raising a LookupError. You can allow these by setting protect to False.

    Certain allowances are made: if a ‘path segment’ is numerical, it’s treated as an array index. If attribute lookup fails, it will re-try on that object using array notation and continue from there. This makes lookup very flexible.

    4.1. Resolving Import References

    The more complete API for name resolution uses the load function, which takes the same optional keyword arguments as traverse. Additionally, this function accepts an optional namespace to search for plugins within. For example:

    from marrow.package.loader import load
    from pip import main
    
    # Load class Foo from example.objects
    load('example.objects:Foo')
    
    # Load the result of the class method ``new`` of the Foo object
    load('example.objects:Foo.new', executable=True)
    
    # Load the "pip" command-line interface.
    assert load('pip', 'console_scripts') is main
    

    Providing a namespace does not prevent explicit object lookup (dot-colon notation) from working.

    4.2. Caching Import References

    An attribute-access dictionary is provided that acts as an import cache:

    from marrow.package.cache import PackageCache
    from pip import main
    
    cache = PackageCache('console_scripts')
    
    assert cache.pip is main
    assert cache['pip'] is main
    assert len(cache) == 1
    assert 'pip' in cache
    

    4.3. Lazy Import Reference Attributes

    You can lazily load and cache an object reference upon dereferencing from an instance using the lazyload utility from the marrow.package.lazy module. Assign the result of calling this function with either an object reference passed in positionally:

    class MyClass:
        debug = lazyload('logging:debug')
    

    Or the attribute path to traverse (using marrow.package.loader:traverse) prefixed by a period:

    class AnotherClass:
        target = 'logging:info'
        log = lazyload('.target')
    

    Any additional arguments are passed to the eventual call to load(). This utility builds on a simpler one that is also offered for fully-tested re-use, lazy, a decorator like @property which will cache the result, with thread-safe locking to ensure only one call will ever be made to the decorated function, per instance.

    5. Managing Plugins

    This package provides two main methods of dealing with plugins and extensions, the first is simple, the second provides full dependency graphing of the extensions.

    5.1. Plugin Manager

    The PluginManager class takes two arguments: the first is the entry point namespace to search, the second is an optional iterable of folders to add to the Python search path for installed packages, allowing your application to have a dedicated plugins folder.

    It provides a register method which take a name and the object to use as the plugin and registers it internally, supporting both attribute and array-like notation for retrieval, as well as iteration of plugins (includes all entry point plugins found and any custom registered ones).

    5.2. Extension Manager

    At a higher level is a PluginManager subclass called ExtensionManager which additionally exposes a sort method capable of resolving dependency order for extensions which follow a simple protocol: have an attribute or array element matching the following, all optional:

    • provides — declare tags describing the features offered by the plugin
    • needs — declare the tags that must be present for this extension to function
    • uses — declare the tags that must be evaluated prior to this extension, but aren’t hard requirements
    • first — declare that this extension is a dependency of all other non-first extensions
    • last — declare that this extension depends on all other non-last extensions
    • excludes — declare tags that must not be present in other plugins for this one to be usable

    6. Version History

    Version 1.0

    • Initial release. Combination of utilities from other Marrow projects.

    Version 1.0.1

    • Extended decorator support. New code paths and tests added to cover canonicalization of decorated functions.

    Version 1.0.2

    • Diagnostic information. Removed extraneous diagnostic information.

    Version 1.1

    • Added lazy evaluation. There are two new helpers for caching of @property-style attributes and lazy lookup of object references.

    Version 1.2

    • Deprecated Python 2.6 and 3.3. While no particular backwards incompatible change was made; as setuptools no longer supports these versions, these versions are now hard/impossible to test.
    • Allow extensions to declare exclusions. Flags that must not be defined for the extension to be usable.

    Version 2.0

    • Updated minimum Python version. Marrow Package now requires Python 3.5 or later.
    • Removed Python 2 support and version specific code. The project has been updated to modern Python packaging standards, including modern namespace use. Modern namespaces are wholly incompatible with the previous namespacing mechanism; this project can not be simultaneously installed with any Marrow project that is Python 2 compatible.
    • Extensive type annotation and in-development validation. When run without optimizations (-O argument to Python or PYTHONOPTIMIZE environment variable) type annotations will be validated.
    • Reduced test fragility. Previously the tests utilized the console_scripts namespace, this was fragile to the presence of other installed libraries, e.g. numpy broke the tests on Travis.

    Version 2.1

    • Migrated from Travis-CI to GitHub Actions for test automation.
    • Implement package-relative path lookup. The load utility function can now resolve the path to a file relative to a package. This is particularly useful for looking up the path to template files or on-disk static assets.
    • Protected attribute access now fails. Underscore-prefixed attributes are assumed to be “protected”, with the technical note that Python adds new internal “double underscore” attributes which must not spontaneously exist, or generate errors other than AttributeError.
    • Tests are now independent of third-party plugin registration.

    Version 2.1.1

    • Update type hinting validation. The typeguard package has removed a functional utility; decoration now used.
    • Canonical plugin name resolution. The name() utility can now resolve the plugin name if given a plugin namespace to check.

    7. License

    Marrow Package has been released under the MIT Open Source license.

    7.1. The MIT License

    Copyright © 2014-2023 Alice Bevan-McGregor and contributors.

    Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

    Visit original content creator repository https://github.com/marrow/package
  • boost-intrusive-pool

    Build Status

    Boost Intrusive Pool

    This project provides a C++ memory pool that is Boost-friendly and performance oriented.

    Features and Limitations

    The boost_intrusive_pool provides the following features:

    • smart pointers: once “allocated” from the pool items whose reference count goes to zero return automatically to the pool;
    • zero-malloc: after a resize of N items, no memory allocations are EVER done until M<=N active items are in use;
    • O(1) allocate;
    • O(1) destroy (pool return);
    • use of standard, well-defined smart pointers: boost::intrusive_ptr<>; see Boost documentation
    • polymorphic-friendly pool: if A derives from boost_intrusive_pool_item, and B derives from A, the memory pool of B works just fine;
    • Header-only;
    • Provides two variants: the infinite memory pool, which automatically enlarges if the number of active items goes over initial memory pool size, and the bounded memory pool, which just returns NULL if trying to allocate more active items than the configured limit;
    • Optional construction via an initialization function: when items are allocated out of the pool via the boost_intrusive_pool::allocate_through_init() API, the init() member function of the memory-pooled objects is called; C++11 perfect forwarding allows to pass optional parameters to the init() routine;
    • Optional construction via custom function: when items are allocated out of the pool via the boost_intrusive_pool::allocate_through_function() the provided custom function is called with the memory-pooled object as argument;
    • Optional recycling via custom function: when the pool is constructed, a custom function std::function can be specified; when items return to the pool it will be called with the item being recycled as parameter; this allows to perform special cleanup like releasing handles, clearing data structures, etc;
    • Optional recycling via alternative function: when items return to the pool, the memory pool can be configured to invoke the destroy() member function of the memory-pooled objects; this allows to perform special cleanup like releasing handles, clearing data structures, etc;

    Of course there are tradeoffs in the design that bring in some limitations:

    • requires all C++ classes stored inside the memory pool to derive from a base class boost_intrusive_pool_item;
    • provides boost::intrusive_ptr<> instead of the more widely-used std::shared_ptr<>: reason is that std::shared_ptr<> puts the reference count in a separate block that needs a separate allocation and thus memory pools based on std::shared_ptr<> (like https://github.com/steinwurf/recycle) cannot be zero-malloc due to the heap-allocated control block;
    • requires C++ classes stored inside the memory pool to have a default constructor: reason is that to ensure the spatial locality of allocated items (for better cache / memory performances) we use the new[] operator which does not allow to provide any parameter;
    • adds about 32 bytes of overhead to each C++ class to be stored inside the memory pool.

    How to Install

    Since this project is header-only it does not need any specific installation, just grab the latest release and put the boost_intrusive_pool.hpp file in your include path.

    Requirements

    This templated memory pool requires a C++11 compliant compiler (it has been tested with GCC 7.x and 8.x). It also requires Boost version 1.55 or higher.

    A Short Tutorial

    The source code in tests/tutorial.cpp provides a short tutorial about the following topics:

    • std::shared_ptr<>
    • boost::intrusive_ptr<>
    • memorypool::boost_intrusive_pool<> (this project)

    and shows that:

    • allocating an item through std::shared_ptr<T> results in the malloc of sizeof(T) plus about 16bytes (the control block);
    • allocating an item through boost::intrusive_ptr<T> results in the malloc of sizeof(T): the refcount is not stored in any separate control block;
    • creation of a memorypool::boost_intrusive_pool<> results in several malloc operations, but then:
    • creation of an item T from a memorypool::boost_intrusive_pool<T> does not result in any malloc

    Example: Using the Default Constructor

    In this example we show how to use memory-pooled objects that are initialized through their default constructor:

    #include "boost_intrusive_pool.hpp"
    
    void main()
    {
    	memorypool::boost_intrusive_pool<DummyClass> pool(4); // allocate pool of size 4
    	   // this results in a big memory allocation; from now on instead it's a zero-malloc world:
    	
    	{
    	    // allocate without ANY call at all (max performances):
    	    boost::intrusive_ptr<DummyClass> hdummy = pool.allocate();
    	
    	    // of course copying pointers around does not trigger any memory allocation:
    	    boost::intrusive_ptr<DummyClass> hdummy2 = hdummy;
    	
    	    // if we observed the pool now it would report: capacity=4, unused_count=3, inuse_count=1
    	    
    	    
    	    // now instead allocate using the DummyClass default ctor:
    	    boost::intrusive_ptr<DummyClass> hdummy3 = pool.allocate_through_init();
    	    
    	} // this time no memory free() will happen!
    
    	// if we observed the pool now it would report: capacity=4, unused_count=4, inuse_count=0
    }
    
    

    Example: Using a non-Default Constructor

    In this example we show how to use memory-pooled objects that are initialized through their NON default constructor:

    #include "boost_intrusive_pool.hpp"
    
    void main()
    {
    	memorypool::boost_intrusive_pool<DummyClass> pool(4); // allocate pool of size 4
    	   // this results in a big memory allocation; from now on instead it's a zero-malloc world:
    	
    	{
    	    // now instead allocate using the DummyClass NON default ctor:
    	    boost::intrusive_ptr<DummyClass> hdummy3 = pool.allocate_through_init(arg1, arg2, arg3);
    	    
    	} // this time no memory free() will happen!
    }
    

    Performance Results

    The following tables show results of some very simple benchmarking obtained on a desktop machine:

    Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz, 4 cores
    16GB DDR3
    libc-2.27 (Ubuntu 18.04)
    

    The memory pool implementation is compared against a “no pool” solution (the plain_malloc line), which simply allocates items directly from the heap through malloc. For both the boost_intrusive_pool and the plain_malloc a very lightweight processing is simulated on the allocated items so that these performance results show the gain you obtain if:

    • you intend to create a memory pool of large items, expensive to allocate each time;
    • the processing for each item is lightweight.

    The benchmarks are then repeated considering 3 different memory allocators:

    1. GNU libc default malloc/free implementation
    2. Google perftools also known as tcmalloc
    3. Jemalloc

    Moreover 2 different memory allocation/deallocation patterns are considered:

    1. Continuous allocations, bulk free at end: all 100 thousands large objects are allocated sequentially, lightweight-processed and then released back to the pool/heap.
    2. Mixed alloc/free pattern: the items are returned to the pool/heap in a pseudo-random order, potentially generating memory fragmentation in the plain_malloc implementation.

    Finally for each combination of memory allocator and allocation pattern we measure the mean time it takes to allocate an item (or retrieve it from the memory pool!) against the “memory pool enlarge step” configuration value. Of course the enlarge step parameter will affect only the memory pool perfomances so that in theory the “plain malloc” time should remain constant (i.e. all green lines should be flat in all graphs below!). In practice small variations are expected also in the measured times for the “plain malloc”.

    Results for the Continuous allocations, bulk free at end benchmark follow:

    Results show that with glibc allocator (regular malloc/free implementation), the use of a memory pool results in up to 44% improvement (from an average of 134ns to about 76ns). When Google’s tcmalloc or jemalloc allocators are in use the improvement grows to 67% and 76% respectively.

    This is important to show that even if your software is using an optimized allocator the memory pool pattern will improve performances considerably.

    Results for the Mixed alloc/free pattern benchmark follow:

    These results show that with a pattern where malloc and free operations are scattered and “randomized” a little bit, regular allocators cannot avoid memory fragmentation and less spatial locality compared to a memory pool implementation.

    In particular improvements go from 40% (glibc) to 53% (jemalloc) and up to 73% (tcmalloc).

    Of course take all these performance results with care. Actual performance gain may vary a lot depending on your rate of malloc/free operations, the pattern in which they happen, the size of the pooled items, etc etc. You can find the source code used to generate these benchmark results in the file tests/performance_tests.cpp.

    About Thread Safety

    The memory pool has no locking mechanism whatsoever and is not thread safe. The reason is that this memory pool is performance oriented and locks are not that fast, specially if you have many threads continuosly allocating and releasing items to the pool. In such a scenario, the best from a performance point of view, is to simply create a memory pool for each thread.

    Other Memory Pools

    This memory pool implementation has been inspired by a few other C++ implementations out there like:

    Visit original content creator repository https://github.com/f18m/boost-intrusive-pool
  • POLENV

                        GNU GENERAL PUBLIC LICENSE
                           Version 3, 29 June 2007
    
     Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
     Everyone is permitted to copy and distribute verbatim copies
     of this license document, but changing it is not allowed.
    
                                Preamble
    
      The GNU General Public License is a free, copyleft license for
    software and other kinds of works.
    
      The licenses for most software and other practical works are designed
    to take away your freedom to share and change the works.  By contrast,
    the GNU General Public License is intended to guarantee your freedom to
    share and change all versions of a program--to make sure it remains free
    software for all its users.  We, the Free Software Foundation, use the
    GNU General Public License for most of our software; it applies also to
    any other work released this way by its authors.  You can apply it to
    your programs, too.
    
      When we speak of free software, we are referring to freedom, not
    price.  Our General Public Licenses are designed to make sure that you
    have the freedom to distribute copies of free software (and charge for
    them if you wish), that you receive source code or can get it if you
    want it, that you can change the software or use pieces of it in new
    free programs, and that you know you can do these things.
    
      To protect your rights, we need to prevent others from denying you
    these rights or asking you to surrender the rights.  Therefore, you have
    certain responsibilities if you distribute copies of the software, or if
    you modify it: responsibilities to respect the freedom of others.
    
      For example, if you distribute copies of such a program, whether
    gratis or for a fee, you must pass on to the recipients the same
    freedoms that you received.  You must make sure that they, too, receive
    or can get the source code.  And you must show them these terms so they
    know their rights.
    
      Developers that use the GNU GPL protect your rights with two steps:
    (1) assert copyright on the software, and (2) offer you this License
    giving you legal permission to copy, distribute and/or modify it.
    
      For the developers' and authors' protection, the GPL clearly explains
    that there is no warranty for this free software.  For both users' and
    authors' sake, the GPL requires that modified versions be marked as
    changed, so that their problems will not be attributed erroneously to
    authors of previous versions.
    
      Some devices are designed to deny users access to install or run
    modified versions of the software inside them, although the manufacturer
    can do so.  This is fundamentally incompatible with the aim of
    protecting users' freedom to change the software.  The systematic
    pattern of such abuse occurs in the area of products for individuals to
    use, which is precisely where it is most unacceptable.  Therefore, we
    have designed this version of the GPL to prohibit the practice for those
    products.  If such problems arise substantially in other domains, we
    stand ready to extend this provision to those domains in future versions
    of the GPL, as needed to protect the freedom of users.
    
      Finally, every program is threatened constantly by software patents.
    States should not allow patents to restrict development and use of
    software on general-purpose computers, but in those that do, we wish to
    avoid the special danger that patents applied to a free program could
    make it effectively proprietary.  To prevent this, the GPL assures that
    patents cannot be used to render the program non-free.
    
      The precise terms and conditions for copying, distribution and
    modification follow.
    
                           TERMS AND CONDITIONS
    
      0. Definitions.
    
      "This License" refers to version 3 of the GNU General Public License.
    
      "Copyright" also means copyright-like laws that apply to other kinds of
    works, such as semiconductor masks.
    
      "The Program" refers to any copyrightable work licensed under this
    License.  Each licensee is addressed as "you".  "Licensees" and
    "recipients" may be individuals or organizations.
    
      To "modify" a work means to copy from or adapt all or part of the work
    in a fashion requiring copyright permission, other than the making of an
    exact copy.  The resulting work is called a "modified version" of the
    earlier work or a work "based on" the earlier work.
    
      A "covered work" means either the unmodified Program or a work based
    on the Program.
    
      To "propagate" a work means to do anything with it that, without
    permission, would make you directly or secondarily liable for
    infringement under applicable copyright law, except executing it on a
    computer or modifying a private copy.  Propagation includes copying,
    distribution (with or without modification), making available to the
    public, and in some countries other activities as well.
    
      To "convey" a work means any kind of propagation that enables other
    parties to make or receive copies.  Mere interaction with a user through
    a computer network, with no transfer of a copy, is not conveying.
    
      An interactive user interface displays "Appropriate Legal Notices"
    to the extent that it includes a convenient and prominently visible
    feature that (1) displays an appropriate copyright notice, and (2)
    tells the user that there is no warranty for the work (except to the
    extent that warranties are provided), that licensees may convey the
    work under this License, and how to view a copy of this License.  If
    the interface presents a list of user commands or options, such as a
    menu, a prominent item in the list meets this criterion.
    
      1. Source Code.
    
      The "source code" for a work means the preferred form of the work
    for making modifications to it.  "Object code" means any non-source
    form of a work.
    
      A "Standard Interface" means an interface that either is an official
    standard defined by a recognized standards body, or, in the case of
    interfaces specified for a particular programming language, one that
    is widely used among developers working in that language.
    
      The "System Libraries" of an executable work include anything, other
    than the work as a whole, that (a) is included in the normal form of
    packaging a Major Component, but which is not part of that Major
    Component, and (b) serves only to enable use of the work with that
    Major Component, or to implement a Standard Interface for which an
    implementation is available to the public in source code form.  A
    "Major Component", in this context, means a major essential component
    (kernel, window system, and so on) of the specific operating system
    (if any) on which the executable work runs, or a compiler used to
    produce the work, or an object code interpreter used to run it.
    
      The "Corresponding Source" for a work in object code form means all
    the source code needed to generate, install, and (for an executable
    work) run the object code and to modify the work, including scripts to
    control those activities.  However, it does not include the work's
    System Libraries, or general-purpose tools or generally available free
    programs which are used unmodified in performing those activities but
    which are not part of the work.  For example, Corresponding Source
    includes interface definition files associated with source files for
    the work, and the source code for shared libraries and dynamically
    linked subprograms that the work is specifically designed to require,
    such as by intimate data communication or control flow between those
    subprograms and other parts of the work.
    
      The Corresponding Source need not include anything that users
    can regenerate automatically from other parts of the Corresponding
    Source.
    
      The Corresponding Source for a work in source code form is that
    same work.
    
      2. Basic Permissions.
    
      All rights granted under this License are granted for the term of
    copyright on the Program, and are irrevocable provided the stated
    conditions are met.  This License explicitly affirms your unlimited
    permission to run the unmodified Program.  The output from running a
    covered work is covered by this License only if the output, given its
    content, constitutes a covered work.  This License acknowledges your
    rights of fair use or other equivalent, as provided by copyright law.
    
      You may make, run and propagate covered works that you do not
    convey, without conditions so long as your license otherwise remains
    in force.  You may convey covered works to others for the sole purpose
    of having them make modifications exclusively for you, or provide you
    with facilities for running those works, provided that you comply with
    the terms of this License in conveying all material for which you do
    not control copyright.  Those thus making or running the covered works
    for you must do so exclusively on your behalf, under your direction
    and control, on terms that prohibit them from making any copies of
    your copyrighted material outside their relationship with you.
    
      Conveying under any other circumstances is permitted solely under
    the conditions stated below.  Sublicensing is not allowed; section 10
    makes it unnecessary.
    
      3. Protecting Users' Legal Rights From Anti-Circumvention Law.
    
      No covered work shall be deemed part of an effective technological
    measure under any applicable law fulfilling obligations under article
    11 of the WIPO copyright treaty adopted on 20 December 1996, or
    similar laws prohibiting or restricting circumvention of such
    measures.
    
      When you convey a covered work, you waive any legal power to forbid
    circumvention of technological measures to the extent such circumvention
    is effected by exercising rights under this License with respect to
    the covered work, and you disclaim any intention to limit operation or
    modification of the work as a means of enforcing, against the work's
    users, your or third parties' legal rights to forbid circumvention of
    technological measures.
    
      4. Conveying Verbatim Copies.
    
      You may convey verbatim copies of the Program's source code as you
    receive it, in any medium, provided that you conspicuously and
    appropriately publish on each copy an appropriate copyright notice;
    keep intact all notices stating that this License and any
    non-permissive terms added in accord with section 7 apply to the code;
    keep intact all notices of the absence of any warranty; and give all
    recipients a copy of this License along with the Program.
    
      You may charge any price or no price for each copy that you convey,
    and you may offer support or warranty protection for a fee.
    
      5. Conveying Modified Source Versions.
    
      You may convey a work based on the Program, or the modifications to
    produce it from the Program, in the form of source code under the
    terms of section 4, provided that you also meet all of these conditions:
    
        a) The work must carry prominent notices stating that you modified
        it, and giving a relevant date.
    
        b) The work must carry prominent notices stating that it is
        released under this License and any conditions added under section
        7.  This requirement modifies the requirement in section 4 to
        "keep intact all notices".
    
        c) You must license the entire work, as a whole, under this
        License to anyone who comes into possession of a copy.  This
        License will therefore apply, along with any applicable section 7
        additional terms, to the whole of the work, and all its parts,
        regardless of how they are packaged.  This License gives no
        permission to license the work in any other way, but it does not
        invalidate such permission if you have separately received it.
    
        d) If the work has interactive user interfaces, each must display
        Appropriate Legal Notices; however, if the Program has interactive
        interfaces that do not display Appropriate Legal Notices, your
        work need not make them do so.
    
      A compilation of a covered work with other separate and independent
    works, which are not by their nature extensions of the covered work,
    and which are not combined with it such as to form a larger program,
    in or on a volume of a storage or distribution medium, is called an
    "aggregate" if the compilation and its resulting copyright are not
    used to limit the access or legal rights of the compilation's users
    beyond what the individual works permit.  Inclusion of a covered work
    in an aggregate does not cause this License to apply to the other
    parts of the aggregate.
    
      6. Conveying Non-Source Forms.
    
      You may convey a covered work in object code form under the terms
    of sections 4 and 5, provided that you also convey the
    machine-readable Corresponding Source under the terms of this License,
    in one of these ways:
    
        a) Convey the object code in, or embodied in, a physical product
        (including a physical distribution medium), accompanied by the
        Corresponding Source fixed on a durable physical medium
        customarily used for software interchange.
    
        b) Convey the object code in, or embodied in, a physical product
        (including a physical distribution medium), accompanied by a
        written offer, valid for at least three years and valid for as
        long as you offer spare parts or customer support for that product
        model, to give anyone who possesses the object code either (1) a
        copy of the Corresponding Source for all the software in the
        product that is covered by this License, on a durable physical
        medium customarily used for software interchange, for a price no
        more than your reasonable cost of physically performing this
        conveying of source, or (2) access to copy the
        Corresponding Source from a network server at no charge.
    
        c) Convey individual copies of the object code with a copy of the
        written offer to provide the Corresponding Source.  This
        alternative is allowed only occasionally and noncommercially, and
        only if you received the object code with such an offer, in accord
        with subsection 6b.
    
        d) Convey the object code by offering access from a designated
        place (gratis or for a charge), and offer equivalent access to the
        Corresponding Source in the same way through the same place at no
        further charge.  You need not require recipients to copy the
        Corresponding Source along with the object code.  If the place to
        copy the object code is a network server, the Corresponding Source
        may be on a different server (operated by you or a third party)
        that supports equivalent copying facilities, provided you maintain
        clear directions next to the object code saying where to find the
        Corresponding Source.  Regardless of what server hosts the
        Corresponding Source, you remain obligated to ensure that it is
        available for as long as needed to satisfy these requirements.
    
        e) Convey the object code using peer-to-peer transmission, provided
        you inform other peers where the object code and Corresponding
        Source of the work are being offered to the general public at no
        charge under subsection 6d.
    
      A separable portion of the object code, whose source code is excluded
    from the Corresponding Source as a System Library, need not be
    included in conveying the object code work.
    
      A "User Product" is either (1) a "consumer product", which means any
    tangible personal property which is normally used for personal, family,
    or household purposes, or (2) anything designed or sold for incorporation
    into a dwelling.  In determining whether a product is a consumer product,
    doubtful cases shall be resolved in favor of coverage.  For a particular
    product received by a particular user, "normally used" refers to a
    typical or common use of that class of product, regardless of the status
    of the particular user or of the way in which the particular user
    actually uses, or expects or is expected to use, the product.  A product
    is a consumer product regardless of whether the product has substantial
    commercial, industrial or non-consumer uses, unless such uses represent
    the only significant mode of use of the product.
    
      "Installation Information" for a User Product means any methods,
    procedures, authorization keys, or other information required to install
    and execute modified versions of a covered work in that User Product from
    a modified version of its Corresponding Source.  The information must
    suffice to ensure that the continued functioning of the modified object
    code is in no case prevented or interfered with solely because
    modification has been made.
    
      If you convey an object code work under this section in, or with, or
    specifically for use in, a User Product, and the conveying occurs as
    part of a transaction in which the right of possession and use of the
    User Product is transferred to the recipient in perpetuity or for a
    fixed term (regardless of how the transaction is characterized), the
    Corresponding Source conveyed under this section must be accompanied
    by the Installation Information.  But this requirement does not apply
    if neither you nor any third party retains the ability to install
    modified object code on the User Product (for example, the work has
    been installed in ROM).
    
      The requirement to provide Installation Information does not include a
    requirement to continue to provide support service, warranty, or updates
    for a work that has been modified or installed by the recipient, or for
    the User Product in which it has been modified or installed.  Access to a
    network may be denied when the modification itself materially and
    adversely affects the operation of the network or violates the rules and
    protocols for communication across the network.
    
      Corresponding Source conveyed, and Installation Information provided,
    in accord with this section must be in a format that is publicly
    documented (and with an implementation available to the public in
    source code form), and must require no special password or key for
    unpacking, reading or copying.
    
      7. Additional Terms.
    
      "Additional permissions" are terms that supplement the terms of this
    License by making exceptions from one or more of its conditions.
    Additional permissions that are applicable to the entire Program shall
    be treated as though they were included in this License, to the extent
    that they are valid under applicable law.  If additional permissions
    apply only to part of the Program, that part may be used separately
    under those permissions, but the entire Program remains governed by
    this License without regard to the additional permissions.
    
      When you convey a copy of a covered work, you may at your option
    remove any additional permissions from that copy, or from any part of
    it.  (Additional permissions may be written to require their own
    removal in certain cases when you modify the work.)  You may place
    additional permissions on material, added by you to a covered work,
    for which you have or can give appropriate copyright permission.
    
      Notwithstanding any other provision of this License, for material you
    add to a covered work, you may (if authorized by the copyright holders of
    that material) supplement the terms of this License with terms:
    
        a) Disclaiming warranty or limiting liability differently from the
        terms of sections 15 and 16 of this License; or
    
        b) Requiring preservation of specified reasonable legal notices or
        author attributions in that material or in the Appropriate Legal
        Notices displayed by works containing it; or
    
        c) Prohibiting misrepresentation of the origin of that material, or
        requiring that modified versions of such material be marked in
        reasonable ways as different from the original version; or
    
        d) Limiting the use for publicity purposes of names of licensors or
        authors of the material; or
    
        e) Declining to grant rights under trademark law for use of some
        trade names, trademarks, or service marks; or
    
        f) Requiring indemnification of licensors and authors of that
        material by anyone who conveys the material (or modified versions of
        it) with contractual assumptions of liability to the recipient, for
        any liability that these contractual assumptions directly impose on
        those licensors and authors.
    
      All other non-permissive additional terms are considered "further
    restrictions" within the meaning of section 10.  If the Program as you
    received it, or any part of it, contains a notice stating that it is
    governed by this License along with a term that is a further
    restriction, you may remove that term.  If a license document contains
    a further restriction but permits relicensing or conveying under this
    License, you may add to a covered work material governed by the terms
    of that license document, provided that the further restriction does
    not survive such relicensing or conveying.
    
      If you add terms to a covered work in accord with this section, you
    must place, in the relevant source files, a statement of the
    additional terms that apply to those files, or a notice indicating
    where to find the applicable terms.
    
      Additional terms, permissive or non-permissive, may be stated in the
    form of a separately written license, or stated as exceptions;
    the above requirements apply either way.
    
      8. Termination.
    
      You may not propagate or modify a covered work except as expressly
    provided under this License.  Any attempt otherwise to propagate or
    modify it is void, and will automatically terminate your rights under
    this License (including any patent licenses granted under the third
    paragraph of section 11).
    
      However, if you cease all violation of this License, then your
    license from a particular copyright holder is reinstated (a)
    provisionally, unless and until the copyright holder explicitly and
    finally terminates your license, and (b) permanently, if the copyright
    holder fails to notify you of the violation by some reasonable means
    prior to 60 days after the cessation.
    
      Moreover, your license from a particular copyright holder is
    reinstated permanently if the copyright holder notifies you of the
    violation by some reasonable means, this is the first time you have
    received notice of violation of this License (for any work) from that
    copyright holder, and you cure the violation prior to 30 days after
    your receipt of the notice.
    
      Termination of your rights under this section does not terminate the
    licenses of parties who have received copies or rights from you under
    this License.  If your rights have been terminated and not permanently
    reinstated, you do not qualify to receive new licenses for the same
    material under section 10.
    
      9. Acceptance Not Required for Having Copies.
    
      You are not required to accept this License in order to receive or
    run a copy of the Program.  Ancillary propagation of a covered work
    occurring solely as a consequence of using peer-to-peer transmission
    to receive a copy likewise does not require acceptance.  However,
    nothing other than this License grants you permission to propagate or
    modify any covered work.  These actions infringe copyright if you do
    not accept this License.  Therefore, by modifying or propagating a
    covered work, you indicate your acceptance of this License to do so.
    
      10. Automatic Licensing of Downstream Recipients.
    
      Each time you convey a covered work, the recipient automatically
    receives a license from the original licensors, to run, modify and
    propagate that work, subject to this License.  You are not responsible
    for enforcing compliance by third parties with this License.
    
      An "entity transaction" is a transaction transferring control of an
    organization, or substantially all assets of one, or subdividing an
    organization, or merging organizations.  If propagation of a covered
    work results from an entity transaction, each party to that
    transaction who receives a copy of the work also receives whatever
    licenses to the work the party's predecessor in interest had or could
    give under the previous paragraph, plus a right to possession of the
    Corresponding Source of the work from the predecessor in interest, if
    the predecessor has it or can get it with reasonable efforts.
    
      You may not impose any further restrictions on the exercise of the
    rights granted or affirmed under this License.  For example, you may
    not impose a license fee, royalty, or other charge for exercise of
    rights granted under this License, and you may not initiate litigation
    (including a cross-claim or counterclaim in a lawsuit) alleging that
    any patent claim is infringed by making, using, selling, offering for
    sale, or importing the Program or any portion of it.
    
      11. Patents.
    
      A "contributor" is a copyright holder who authorizes use under this
    License of the Program or a work on which the Program is based.  The
    work thus licensed is called the contributor's "contributor version".
    
      A contributor's "essential patent claims" are all patent claims
    owned or controlled by the contributor, whether already acquired or
    hereafter acquired, that would be infringed by some manner, permitted
    by this License, of making, using, or selling its contributor version,
    but do not include claims that would be infringed only as a
    consequence of further modification of the contributor version.  For
    purposes of this definition, "control" includes the right to grant
    patent sublicenses in a manner consistent with the requirements of
    this License.
    
      Each contributor grants you a non-exclusive, worldwide, royalty-free
    patent license under the contributor's essential patent claims, to
    make, use, sell, offer for sale, import and otherwise run, modify and
    propagate the contents of its contributor version.
    
      In the following three paragraphs, a "patent license" is any express
    agreement or commitment, however denominated, not to enforce a patent
    (such as an express permission to practice a patent or covenant not to
    sue for patent infringement).  To "grant" such a patent license to a
    party means to make such an agreement or commitment not to enforce a
    patent against the party.
    
      If you convey a covered work, knowingly relying on a patent license,
    and the Corresponding Source of the work is not available for anyone
    to copy, free of charge and under the terms of this License, through a
    publicly available network server or other readily accessible means,
    then you must either (1) cause the Corresponding Source to be so
    available, or (2) arrange to deprive yourself of the benefit of the
    patent license for this particular work, or (3) arrange, in a manner
    consistent with the requirements of this License, to extend the patent
    license to downstream recipients.  "Knowingly relying" means you have
    actual knowledge that, but for the patent license, your conveying the
    covered work in a country, or your recipient's use of the covered work
    in a country, would infringe one or more identifiable patents in that
    country that you have reason to believe are valid.
    
      If, pursuant to or in connection with a single transaction or
    arrangement, you convey, or propagate by procuring conveyance of, a
    covered work, and grant a patent license to some of the parties
    receiving the covered work authorizing them to use, propagate, modify
    or convey a specific copy of the covered work, then the patent license
    you grant is automatically extended to all recipients of the covered
    work and works based on it.
    
      A patent license is "discriminatory" if it does not include within
    the scope of its coverage, prohibits the exercise of, or is
    conditioned on the non-exercise of one or more of the rights that are
    specifically granted under this License.  You may not convey a covered
    work if you are a party to an arrangement with a third party that is
    in the business of distributing software, under which you make payment
    to the third party based on the extent of your activity of conveying
    the work, and under which the third party grants, to any of the
    parties who would receive the covered work from you, a discriminatory
    patent license (a) in connection with copies of the covered work
    conveyed by you (or copies made from those copies), or (b) primarily
    for and in connection with specific products or compilations that
    contain the covered work, unless you entered into that arrangement,
    or that patent license was granted, prior to 28 March 2007.
    
      Nothing in this License shall be construed as excluding or limiting
    any implied license or other defenses to infringement that may
    otherwise be available to you under applicable patent law.
    
      12. No Surrender of Others' Freedom.
    
      If conditions are imposed on you (whether by court order, agreement or
    otherwise) that contradict the conditions of this License, they do not
    excuse you from the conditions of this License.  If you cannot convey a
    covered work so as to satisfy simultaneously your obligations under this
    License and any other pertinent obligations, then as a consequence you may
    not convey it at all.  For example, if you agree to terms that obligate you
    to collect a royalty for further conveying from those to whom you convey
    the Program, the only way you could satisfy both those terms and this
    License would be to refrain entirely from conveying the Program.
    
      13. Use with the GNU Affero General Public License.
    
      Notwithstanding any other provision of this License, you have
    permission to link or combine any covered work with a work licensed
    under version 3 of the GNU Affero General Public License into a single
    combined work, and to convey the resulting work.  The terms of this
    License will continue to apply to the part which is the covered work,
    but the special requirements of the GNU Affero General Public License,
    section 13, concerning interaction through a network will apply to the
    combination as such.
    
      14. Revised Versions of this License.
    
      The Free Software Foundation may publish revised and/or new versions of
    the GNU General Public License from time to time.  Such new versions will
    be similar in spirit to the present version, but may differ in detail to
    address new problems or concerns.
    
      Each version is given a distinguishing version number.  If the
    Program specifies that a certain numbered version of the GNU General
    Public License "or any later version" applies to it, you have the
    option of following the terms and conditions either of that numbered
    version or of any later version published by the Free Software
    Foundation.  If the Program does not specify a version number of the
    GNU General Public License, you may choose any version ever published
    by the Free Software Foundation.
    
      If the Program specifies that a proxy can decide which future
    versions of the GNU General Public License can be used, that proxy's
    public statement of acceptance of a version permanently authorizes you
    to choose that version for the Program.
    
      Later license versions may give you additional or different
    permissions.  However, no additional obligations are imposed on any
    author or copyright holder as a result of your choosing to follow a
    later version.
    
      15. Disclaimer of Warranty.
    
      THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
    APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
    HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
    OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
    THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
    PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
    IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
    ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
    
      16. Limitation of Liability.
    
      IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
    WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
    THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
    GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
    USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
    DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
    PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
    EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGES.
    
      17. Interpretation of Sections 15 and 16.
    
      If the disclaimer of warranty and limitation of liability provided
    above cannot be given local legal effect according to their terms,
    reviewing courts shall apply local law that most closely approximates
    an absolute waiver of all civil liability in connection with the
    Program, unless a warranty or assumption of liability accompanies a
    copy of the Program in return for a fee.
    
                         END OF TERMS AND CONDITIONS
    
                How to Apply These Terms to Your New Programs
    
      If you develop a new program, and you want it to be of the greatest
    possible use to the public, the best way to achieve this is to make it
    free software which everyone can redistribute and change under these terms.
    
      To do so, attach the following notices to the program.  It is safest
    to attach them to the start of each source file to most effectively
    state the exclusion of warranty; and each file should have at least
    the "copyright" line and a pointer to where the full notice is found.
    
        <one line to give the program's name and a brief idea of what it does.>
        Copyright (C) <year>  <name of author>
    
        This program is free software: you can redistribute it and/or modify
        it under the terms of the GNU General Public License as published by
        the Free Software Foundation, either version 3 of the License, or
        (at your option) any later version.
    
        This program is distributed in the hope that it will be useful,
        but WITHOUT ANY WARRANTY; without even the implied warranty of
        MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
        GNU General Public License for more details.
    
        You should have received a copy of the GNU General Public License
        along with this program.  If not, see <http://www.gnu.org/licenses/>.
    
    Also add information on how to contact you by electronic and paper mail.
    
      If the program does terminal interaction, make it output a short
    notice like this when it starts in an interactive mode:
    
        <program>  Copyright (C) <year>  <name of author>
        This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
        This is free software, and you are welcome to redistribute it
        under certain conditions; type `show c' for details.
    
    The hypothetical commands `show w' and `show c' should show the appropriate
    parts of the General Public License.  Of course, your program's commands
    might be different; for a GUI interface, you would use an "about box".
    
      You should also get your employer (if you work as a programmer) or school,
    if any, to sign a "copyright disclaimer" for the program, if necessary.
    For more information on this, and how to apply and follow the GNU GPL, see
    <http://www.gnu.org/licenses/>.
    
      The GNU General Public License does not permit incorporating your program
    into proprietary programs.  If your program is a subroutine library, you
    may consider it more useful to permit linking proprietary applications with
    the library.  If this is what you want to do, use the GNU Lesser General
    Public License instead of this License.  But first, please read
    <http://www.gnu.org/philosophy/why-not-lgpl.html>.
    

    Visit original content creator repository
    https://github.com/Libaud/POLENV

  • movies-analysis

    Creative Commons Legal Code
    
    CC0 1.0 Universal
    
        CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
        LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
        ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
        INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
        REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
        PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
        THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
        HEREUNDER.
    
    Statement of Purpose
    
    The laws of most jurisdictions throughout the world automatically confer
    exclusive Copyright and Related Rights (defined below) upon the creator
    and subsequent owner(s) (each and all, an "owner") of an original work of
    authorship and/or a database (each, a "Work").
    
    Certain owners wish to permanently relinquish those rights to a Work for
    the purpose of contributing to a commons of creative, cultural and
    scientific works ("Commons") that the public can reliably and without fear
    of later claims of infringement build upon, modify, incorporate in other
    works, reuse and redistribute as freely as possible in any form whatsoever
    and for any purposes, including without limitation commercial purposes.
    These owners may contribute to the Commons to promote the ideal of a free
    culture and the further production of creative, cultural and scientific
    works, or to gain reputation or greater distribution for their Work in
    part through the use and efforts of others.
    
    For these and/or other purposes and motivations, and without any
    expectation of additional consideration or compensation, the person
    associating CC0 with a Work (the "Affirmer"), to the extent that he or she
    is an owner of Copyright and Related Rights in the Work, voluntarily
    elects to apply CC0 to the Work and publicly distribute the Work under its
    terms, with knowledge of his or her Copyright and Related Rights in the
    Work and the meaning and intended legal effect of CC0 on those rights.
    
    1. Copyright and Related Rights. A Work made available under CC0 may be
    protected by copyright and related or neighboring rights ("Copyright and
    Related Rights"). Copyright and Related Rights include, but are not
    limited to, the following:
    
      i. the right to reproduce, adapt, distribute, perform, display,
         communicate, and translate a Work;
     ii. moral rights retained by the original author(s) and/or performer(s);
    iii. publicity and privacy rights pertaining to a person's image or
         likeness depicted in a Work;
     iv. rights protecting against unfair competition in regards to a Work,
         subject to the limitations in paragraph 4(a), below;
      v. rights protecting the extraction, dissemination, use and reuse of data
         in a Work;
     vi. database rights (such as those arising under Directive 96/9/EC of the
         European Parliament and of the Council of 11 March 1996 on the legal
         protection of databases, and under any national implementation
         thereof, including any amended or successor version of such
         directive); and
    vii. other similar, equivalent or corresponding rights throughout the
         world based on applicable law or treaty, and any national
         implementations thereof.
    
    2. Waiver. To the greatest extent permitted by, but not in contravention
    of, applicable law, Affirmer hereby overtly, fully, permanently,
    irrevocably and unconditionally waives, abandons, and surrenders all of
    Affirmer's Copyright and Related Rights and associated claims and causes
    of action, whether now known or unknown (including existing as well as
    future claims and causes of action), in the Work (i) in all territories
    worldwide, (ii) for the maximum duration provided by applicable law or
    treaty (including future time extensions), (iii) in any current or future
    medium and for any number of copies, and (iv) for any purpose whatsoever,
    including without limitation commercial, advertising or promotional
    purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
    member of the public at large and to the detriment of Affirmer's heirs and
    successors, fully intending that such Waiver shall not be subject to
    revocation, rescission, cancellation, termination, or any other legal or
    equitable action to disrupt the quiet enjoyment of the Work by the public
    as contemplated by Affirmer's express Statement of Purpose.
    
    3. Public License Fallback. Should any part of the Waiver for any reason
    be judged legally invalid or ineffective under applicable law, then the
    Waiver shall be preserved to the maximum extent permitted taking into
    account Affirmer's express Statement of Purpose. In addition, to the
    extent the Waiver is so judged Affirmer hereby grants to each affected
    person a royalty-free, non transferable, non sublicensable, non exclusive,
    irrevocable and unconditional license to exercise Affirmer's Copyright and
    Related Rights in the Work (i) in all territories worldwide, (ii) for the
    maximum duration provided by applicable law or treaty (including future
    time extensions), (iii) in any current or future medium and for any number
    of copies, and (iv) for any purpose whatsoever, including without
    limitation commercial, advertising or promotional purposes (the
    "License"). The License shall be deemed effective as of the date CC0 was
    applied by Affirmer to the Work. Should any part of the License for any
    reason be judged legally invalid or ineffective under applicable law, such
    partial invalidity or ineffectiveness shall not invalidate the remainder
    of the License, and in such case Affirmer hereby affirms that he or she
    will not (i) exercise any of his or her remaining Copyright and Related
    Rights in the Work or (ii) assert any associated claims and causes of
    action with respect to the Work, in either case contrary to Affirmer's
    express Statement of Purpose.
    
    4. Limitations and Disclaimers.
    
     a. No trademark or patent rights held by Affirmer are waived, abandoned,
        surrendered, licensed or otherwise affected by this document.
     b. Affirmer offers the Work as-is and makes no representations or
        warranties of any kind concerning the Work, express, implied,
        statutory or otherwise, including without limitation warranties of
        title, merchantability, fitness for a particular purpose, non
        infringement, or the absence of latent or other defects, accuracy, or
        the present or absence of errors, whether or not discoverable, all to
        the greatest extent permissible under applicable law.
     c. Affirmer disclaims responsibility for clearing rights of other persons
        that may apply to the Work or any use thereof, including without
        limitation any person's Copyright and Related Rights in the Work.
        Further, Affirmer disclaims responsibility for obtaining any necessary
        consents, permissions or other rights required for any use of the
        Work.
     d. Affirmer understands and acknowledges that Creative Commons is not a
        party to this document and has no duty or obligation with respect to
        this CC0 or use of the Work.
    

    Visit original content creator repository
    https://github.com/sina-programer/movies-analysis

  • puppet-filemapper

    FileMapper module for Puppet

    Build Status Code Coverage Puppet Forge Puppet Forge - downloads Puppet Forge - endorsement Puppet Forge - scores

    Synopsis

    Map files to resources and back with this handy dandy mixin!

    Description

    Things that are harder than they should be:

    • Acquiring a pet monkey
    • Getting anywhere in Los Angeles
    • Understanding the ParsedFile provider
    • Writing Puppet providers that directly manipulate files

    The solution for this is to completely bypass parsing in any sort of base provider, and delegate the role of parsing and generating to including classes.

    You figure out how to parse and write the file, and this will do the rest.

    Synopsis of implementation requirements

    Providers using the Filemapper extension need to implement the following methods.

    self.target_files

    This should return an array of filenames specifying which files should be prefetched.

    self.parse_file(filename, file_contents)

    This should take two values, a string containing the file name, and a string containing the contents of the file. It should return an array of hashes, where each hash represents {property => value} pairs.

    select_file

    This is a provider instance method. It should return a string containing the filename that the provider should be flushed to.

    self.format_file(filename, providers)

    This should take two values, a string containing the file name to be flushed, and an array of providers that should be flushed to this file. It should return a string containing the contents of the file to be written.

    Synopsis of optional implementation hooks

    self.pre_flush_hook(filename) and self.post_flush_hook(filename)

    These methods can be implemented to add behavior right before and right after filesystem operations. Both methods take a single argument, a string containing the name of the file to be flushed.

    If self.pre_flush_hook raises an exception, the flush will not occur and the provider will be marked as failed and will refuse to perform any more flushes. If some sort of critical error occurred, this can force the provider to error out before it starts stomping on files.

    self.post_flush_hook is guaranteed to run after any filesystem operations occur. This can be used for recovery if something goes wrong during the flush. If this method raises an exception, the provider will be marked as failed and will refuse to perform any more flushes.

    Removing empty files

    If a file is empty, it’s often reasonable to just delete it. The Filemapper mixin implements attr_accessor :unlink_empty_files. If that value is set to true, then if self.format_file returns the empty string then the file will be deleted from the file system.

    How it works

    The Filemapper extension takes advantage of hooks within the Transaction to reduce the number of reads and writes needed to perform operations.

    prefetching

    When a catalog is being applied, providers can define the prefetch method to load all resources before runtime. The Filemapper extension uses this method to preemptively read all files that the provider requires, and generates and stores the state of the requested resources. This means that if you have a few thousand resources in 20 files, you only need to do 20 reads for the entire Puppet run.

    post-evaluation flushing

    When resources are normally evaluated, each time a property is synchronized it’s expected that an action will be run right then. The Filemapper extension instead records all the requested changes and defers operating on them. When the resource is finished, it will be flushed, at which time all of the requested changes will be applied in one pass. Given a resource with 10 properties, all of which are out of sync, the file will be written only once. If no properties are out of sync, the file will be untouched.

    To ensure that the system state matches what Puppet thinks is going on, any file that has changed resources will be re-written after each resource is flushed. That means that if you have 20 resources out of sync, that file will have to be written 20 times. While it’s technically possible to write the file in a single pass, this means that some resources will be applied either early or late, which utterly smashes POLA.

    Use on the command line

    The Filemapper extension implements the instances method, which means that you can use the puppet resource command to interact with the associated provider without having to perform a full blown Puppet run.

    Selecting files to load

    In order to provide prefetching and puppet resource in a clean manner, the Filemapper extension has to have a full list of what files to read. Implementing classes need to implement the target_files method which returns a list of files to read. The implementation is entirely up to the implementing class; it can return a single file every time, such as “/etc/inittab”, or it can generate that information on the fly, by returning Dir["/etc/sysconfig/network/ifcfg-*"]. Basically, files that will be used as a source of data can be as complex or simple as you need.

    Writing back files

    In a similar vein, resources can be written back to files in whatever method you need. Implementing classes need to implement the instance method #select_file so that when that resource is changed, the correct file is modified.

    Parsing

    When parsing a file, the implementing class needs to implement the parse_file method. It will get the name of the file being parsed as well as the contents. It can parse this file in whatever manner needed, and should return an array of any provider instances generated. If the file only contains a single provider instance, then just wrap that instance in an array.

    Writing

    Whenever a file is marked as dirty, that is a resource associated with that file has changed, the format_file method will be called. The implementing class needs to implement a method that takes the filename and an array of provider instances associated with that file, and return a string. The method needs to determine how that file should be written to disk and then return the contents. This can be as complex as needed.

    Under no conditions should implementing classes modify any files directly. No, seriously, don’t do it. The Filemapper extension uses the built in methods for modifying files, which will back up changed files to the filebucket. This is for your own safety, so if you bypass this then you are on your own.

    Storing state outside of resources

    It’s more or less expected that there will be no state outside of the provider instances, but there are plenty of cases where this could be the case. For instance, if one wanted to preserve the comments in a file but didn’t directly associate them with resource attributes, the parse_file method can store data in an instance variable, such as @comments = my_list_of_comments. When formatting the file, the implementing class can read the @comments variable and re-add that data to the content that will be written back.

    Basically, you can store whatever data you need in these methods and pass things around to maintain more complex state.

    Using this sort of operation of reading outside state, you can theoretically have multiple Filemapper extensions that work on shared files. By communicating the state between them, you can manage multiple different resources in one file. HOWEVER, this will require careful communication, so don’t take this sort of thing lightly. However, I don’t thing that anything else in Puppet can provide this sort of behavior. YMMV.

    Why a mixin?

    While the ParsedFile provider is supposed to be inherited, this class is a mixin and needs to be included. This is done because the Filemapper extension only adds behavior, and isn’t really an object or entity in its own right. This way you can use the Filemapper extension while inheriting from something like the Puppet::Provider::Package provider.

    The Backstory

    Managing Unix-ish systems generally means dealing with one of two things:

    1. Processes – starting them, stopping them, monitoring them, etc.
    2. Files – Creating them, editing, deleting them, specifying permissions, etc.

    Puppet has pretty good support in the provider layer for running commands, but the file manipulation layer has been lacking. The long-standing approach for manipulating files has been to select one of the following, and hope for the best.

    Shipping flat files to the client

    Using the File resource to ship flat files is a really common solution, and it’s very easy. It also has the finesse of a brick thrown through a window. There is very little customizability here, aside from the array notation for specifying the source field.

    Using ERB templates to customize files

    The File resource can also take a content field, to which you can pass the output of a template. This allows more sophistication, but not much. It also adds more of a burden to your master; template rendering happens on the master and if you’re doing really crazy number crunching then this pain will be centralized.

    Using Augeas

    Augeas is a very powerful tool that allows you to manipulate files, and the Augeas type allows you to harness this inside of Puppet. However, it has a rather byzantine syntax, and is dependent on lenses being available.

    Sed

    I personally love sed, but sed a file configuration management tool is not.

    Using the ParsedFile provider

    Puppet has a provider extension called the ParsedFile provider that’s used to manipulate text like crontabs and so forth. It also uses a number of advanced features in puppet, which makes it quite powerful. However, it’s incredibly complex, tightly coupled with the FileParsing utility language, has tons of obscure and undocumented hooks that are the only way to do complex operations, and is entirely record based which makes it unsuitable for managing files that have complex structure. While it has basic support for managing multiple files, basic is the indicative word.


    The Filemapper extension has been designed as a lower level alternative to the ParsedFile.

    Examples

    The Filemapper extension was largely extracted out of the puppet-network module. That code base should display the weird edge cases that this extension handles.

    Visit original content creator repository https://github.com/voxpupuli/puppet-filemapper
  • puppet-filemapper

    FileMapper module for Puppet

    Build Status Code Coverage Puppet Forge Puppet Forge - downloads Puppet Forge - endorsement Puppet Forge - scores

    Synopsis

    Map files to resources and back with this handy dandy mixin!

    Description

    Things that are harder than they should be:

    • Acquiring a pet monkey
    • Getting anywhere in Los Angeles
    • Understanding the ParsedFile provider
    • Writing Puppet providers that directly manipulate files

    The solution for this is to completely bypass parsing in any sort of base provider, and delegate the role of parsing and generating to including classes.

    You figure out how to parse and write the file, and this will do the rest.

    Synopsis of implementation requirements

    Providers using the Filemapper extension need to implement the following methods.

    self.target_files

    This should return an array of filenames specifying which files should be prefetched.

    self.parse_file(filename, file_contents)

    This should take two values, a string containing the file name, and a string containing the contents of the file. It should return an array of hashes, where each hash represents {property => value} pairs.

    select_file

    This is a provider instance method. It should return a string containing the filename that the provider should be flushed to.

    self.format_file(filename, providers)

    This should take two values, a string containing the file name to be flushed, and an array of providers that should be flushed to this file. It should return a string containing the contents of the file to be written.

    Synopsis of optional implementation hooks

    self.pre_flush_hook(filename) and self.post_flush_hook(filename)

    These methods can be implemented to add behavior right before and right after filesystem operations. Both methods take a single argument, a string containing the name of the file to be flushed.

    If self.pre_flush_hook raises an exception, the flush will not occur and the provider will be marked as failed and will refuse to perform any more flushes. If some sort of critical error occurred, this can force the provider to error out before it starts stomping on files.

    self.post_flush_hook is guaranteed to run after any filesystem operations occur. This can be used for recovery if something goes wrong during the flush. If this method raises an exception, the provider will be marked as failed and will refuse to perform any more flushes.

    Removing empty files

    If a file is empty, it’s often reasonable to just delete it. The Filemapper mixin implements attr_accessor :unlink_empty_files. If that value is set to true, then if self.format_file returns the empty string then the file will be deleted from the file system.

    How it works

    The Filemapper extension takes advantage of hooks within the Transaction to reduce the number of reads and writes needed to perform operations.

    prefetching

    When a catalog is being applied, providers can define the prefetch method to load all resources before runtime. The Filemapper extension uses this method to preemptively read all files that the provider requires, and generates and stores the state of the requested resources. This means that if you have a few thousand resources in 20 files, you only need to do 20 reads for the entire Puppet run.

    post-evaluation flushing

    When resources are normally evaluated, each time a property is synchronized it’s expected that an action will be run right then. The Filemapper extension instead records all the requested changes and defers operating on them. When the resource is finished, it will be flushed, at which time all of the requested changes will be applied in one pass. Given a resource with 10 properties, all of which are out of sync, the file will be written only once. If no properties are out of sync, the file will be untouched.

    To ensure that the system state matches what Puppet thinks is going on, any file that has changed resources will be re-written after each resource is flushed. That means that if you have 20 resources out of sync, that file will have to be written 20 times. While it’s technically possible to write the file in a single pass, this means that some resources will be applied either early or late, which utterly smashes POLA.

    Use on the command line

    The Filemapper extension implements the instances method, which means that you can use the puppet resource command to interact with the associated provider without having to perform a full blown Puppet run.

    Selecting files to load

    In order to provide prefetching and puppet resource in a clean manner, the Filemapper extension has to have a full list of what files to read. Implementing classes need to implement the target_files method which returns a list of files to read. The implementation is entirely up to the implementing class; it can return a single file every time, such as “/etc/inittab”, or it can generate that information on the fly, by returning Dir["/etc/sysconfig/network/ifcfg-*"]. Basically, files that will be used as a source of data can be as complex or simple as you need.

    Writing back files

    In a similar vein, resources can be written back to files in whatever method you need. Implementing classes need to implement the instance method #select_file so that when that resource is changed, the correct file is modified.

    Parsing

    When parsing a file, the implementing class needs to implement the parse_file method. It will get the name of the file being parsed as well as the contents. It can parse this file in whatever manner needed, and should return an array of any provider instances generated. If the file only contains a single provider instance, then just wrap that instance in an array.

    Writing

    Whenever a file is marked as dirty, that is a resource associated with that file has changed, the format_file method will be called. The implementing class needs to implement a method that takes the filename and an array of provider instances associated with that file, and return a string. The method needs to determine how that file should be written to disk and then return the contents. This can be as complex as needed.

    Under no conditions should implementing classes modify any files directly. No, seriously, don’t do it. The Filemapper extension uses the built in methods for modifying files, which will back up changed files to the filebucket. This is for your own safety, so if you bypass this then you are on your own.

    Storing state outside of resources

    It’s more or less expected that there will be no state outside of the provider instances, but there are plenty of cases where this could be the case. For instance, if one wanted to preserve the comments in a file but didn’t directly associate them with resource attributes, the parse_file method can store data in an instance variable, such as @comments = my_list_of_comments. When formatting the file, the implementing class can read the @comments variable and re-add that data to the content that will be written back.

    Basically, you can store whatever data you need in these methods and pass things around to maintain more complex state.

    Using this sort of operation of reading outside state, you can theoretically have multiple Filemapper extensions that work on shared files. By communicating the state between them, you can manage multiple different resources in one file. HOWEVER, this will require careful communication, so don’t take this sort of thing lightly. However, I don’t thing that anything else in Puppet can provide this sort of behavior. YMMV.

    Why a mixin?

    While the ParsedFile provider is supposed to be inherited, this class is a mixin and needs to be included. This is done because the Filemapper extension only adds behavior, and isn’t really an object or entity in its own right. This way you can use the Filemapper extension while inheriting from something like the Puppet::Provider::Package provider.

    The Backstory

    Managing Unix-ish systems generally means dealing with one of two things:

    1. Processes – starting them, stopping them, monitoring them, etc.
    2. Files – Creating them, editing, deleting them, specifying permissions, etc.

    Puppet has pretty good support in the provider layer for running commands, but the file manipulation layer has been lacking. The long-standing approach for manipulating files has been to select one of the following, and hope for the best.

    Shipping flat files to the client

    Using the File resource to ship flat files is a really common solution, and it’s very easy. It also has the finesse of a brick thrown through a window. There is very little customizability here, aside from the array notation for specifying the source field.

    Using ERB templates to customize files

    The File resource can also take a content field, to which you can pass the output of a template. This allows more sophistication, but not much. It also adds more of a burden to your master; template rendering happens on the master and if you’re doing really crazy number crunching then this pain will be centralized.

    Using Augeas

    Augeas is a very powerful tool that allows you to manipulate files, and the Augeas type allows you to harness this inside of Puppet. However, it has a rather byzantine syntax, and is dependent on lenses being available.

    Sed

    I personally love sed, but sed a file configuration management tool is not.

    Using the ParsedFile provider

    Puppet has a provider extension called the ParsedFile provider that’s used to manipulate text like crontabs and so forth. It also uses a number of advanced features in puppet, which makes it quite powerful. However, it’s incredibly complex, tightly coupled with the FileParsing utility language, has tons of obscure and undocumented hooks that are the only way to do complex operations, and is entirely record based which makes it unsuitable for managing files that have complex structure. While it has basic support for managing multiple files, basic is the indicative word.


    The Filemapper extension has been designed as a lower level alternative to the ParsedFile.

    Examples

    The Filemapper extension was largely extracted out of the puppet-network module. That code base should display the weird edge cases that this extension handles.

    Visit original content creator repository https://github.com/voxpupuli/puppet-filemapper