Avro

Avro:

its a language neutral-data serialization system.
data is described using a language-independent schema.
code generation is optional here, because reading and writing data that conforms to a  given schema even if your code has not seen that particular schema before. Avro assumes that the schema is always
present at the both read and write time which makes for a very compact encoding, since encoded values do not need to be tagged with a field identifier.

Data Serialization:
in the context of the data storage, serialization is a process of translating data structures or object state into the format such that it can be stored(in the file, or buffer, or transmitted across network connection link) and reconstructed later inthe same or another computer environment.

Avro created to address the downside of the hadoop writables that is language portability.
- it precisely defines the binary format.
- Avro schema's are usually written in JSON and the data usually encoded using a binary format but there are other options too.
- the schema used to read the data need not to be same as the schema used for the write a data

Avro specifies an object container format for sequence of the objects. An Avro data file has a metadata section where the schema is stored which makes the file self-describing. These data files support compression and splittable which is crucial for MapReduce data input format.

Pig, Hive, Crunch, Spark can read and write Avro Data files.
Avro has data types and schema:
Primitive Data types:
null
boolean
int
long
float
double
bytes
string
Complex Data Types:
Array  - An ordered collection of objects. All objects in a particular array must have the same schema

{
  "type" : "array",
  "items":"long"
}

Map: an unordered collection of key value pairs. keys must be strings and values may be any type, although with a particular may all values must have the same schema.
{
  "type": "map",
   "values":"string"
}

Record: collection of the named fields of any type:
{
 "type":"record",
 "name" : "weather record",
 "doc":"a weather reading",
 "fields" : [{"name":"year","type":"int"},
                 {"name":"temperature","type":"int"},
                 {"name":"stationid","type":"string"}
                ]
}

Enum: Aset of named values
{
 "type":"enum",
 "name":"cutlery",
 "doc":"An eating utensil",
 "symbol":["knife","fork","spoon"]
}

Fixed: a fixed number of 8 bits unsigned bytes
{
"type":"fixed",
"name":"marchas",
"size":16
}


Union: a union of the schema's. A union is represented by json array , where each elements in the array is schema. data represented by the union must match one of the schema's in the union.
when a schema is not known ahead of the run time dynamic mapping is used. it is similar to java generic mapping.

Specific mapping or generic mapping:

java has a third mapping , the reflect mapping, which maps Avro types onto persisting java types which uses reflection.
Avro string can be represented by either java string or Avro UTF8 Java type. because UTF8 for efficiency. Single UTF8 instance may be reused for reading or writing a series of the values. Java string decodes UTF-8 object construction time, Avro does it lazily which can increase the performance in same case.