1^ 'P`!ů `Y~"ά_r-_x f_D$ ;6cF"0Ɇi/|:ıC"
pܭ[TS ѽ{wkcp뭷kXQgydAD Y 6 UXXg
`y |N&N0(**ny =iiiHф jԔoEd ! 8Ξv-337$3=|
`Xqql *6mrKhrS.]lؾ}{,Spc}ƘTj oz
`"""""""s4+ |A \{LqqqB( 1# uFܑWD2eJB5DU9oǴ
n"2iҤI1M.`j
o~-jk;=ۉ %. nZ aiI+;lWPP 3 ,Բ5yg'///`>@TU[CK"(G(G~Y ϖ G0 ; >+"
.y"e F8e _ ]ɨm۶UWWOuuh۶mstq6iON}>DX<֑ ND͞=Ԫ \u j &"""""" K"m9]UG 81k`(",SEEEPLDzcNQ
شTƘW1Bw7 owA([ƘTbfHeb(,0aBM JUOa:w& 2eJAHcוwXo?GF? .6ƼH$_ `. GƮ
Mֹ/> zvo; l- *g_N" @ cSծ :#[6ށ!cz;/nVD??eee a>>7[!9 v _cVrss?7n\7鉜'" ucnx $Ѩ] _{5x㍎o㍎V__A^ Çoj_1پp+))
u(fN;Ȕ)S6#|?ΒDdw yMPZZڄƟ3|M ѹ1i #iiihݺ5v(v gۺ` Q\\ٳg @י(LDDDDDDDDDk γ# MM67xɯڷ#` C
>7sNꪜqyӞ6XKR !0ɁC
f# `
FFF5 ~-:osV^?J2ْӴvJ&w G-ZR)]gy\ήC0uUUI Rw}%9iHH)ZkW%Y`W OOrΖܱ&%& fxSZk.];3k^{핝wyr$HrZ) 1ɯ
va? b=v p
Krv$gw>5ɕ]fxg[3QJy[)ڮsl~hFJ) r7ZkA)RI&$? ɉ{;=c%Ҁ$ɾg=Y $*ivzd?rK:Gn 0E~$AZ{G:z5?%b96Z`sL|,Z;.v?* p!]Vk=w5H)Rʛ<%)I9Sty)sC7ޟwoK
## master
Nothing yet
## 1.1.1 - 2022-03-03
### Fixed
- Fixed warning when generating autoload classmap via composer.
## 1.1.0 - 2022-02-19
### Added
- Autoloading without Composer. Thanks @a-sync.
## 1.0.1 - 2022-02-06
### Fixed
- Broken command `make performance-tests`
- Slight performance improvements
## 1.0.0 - 2022-02-04
### Removed
- Removed deprecated functions `objects()` and `httpClientChunks()`.
- Removed deprecated `JsonMachine` entrypoint class. Use `Items` instead.
- Removed deprecated `Decoder` interface. Use `ItemDecoder` instead.
- Removed `Parser::getJsonPointer()`. Use `Parser::getJsonPointers()`/`Items::getJsonPointers()` instead.
- Removed `Parser::getJsonPointerPath()`. No replacement. Was not useful for anything other than testing and exposed internal implementation.
### Changed
#### Simplified and fixed decoding
- JSON Pointer parts between slashes (a.k.a reference tokens) must be valid encoded JSON strings to be [JSON Pointer RFC 6901](https://tools.ietf.org/html/rfc6901) compliant.
It means that no internal key decoding is performed anymore. You will have to change your JSON Pointers if you match against keys with escape sequences.
```diff
Items::fromString(
'{"quotes\"": [1, 2, 3]}',
- ['pointer' => '/quotes"']
+ ['pointer' => '/quotes\"']
);
```
- Method `ItemDecoder::decodeInternalKey()` was deleted as well as related `ValidStringResult`.
They are not used anymore as described in previous point.
- `PassThruDecoder` does not decode keys anymore. Both the key and the value yielded are raw JSON now.
#### Other
- Default decoding structure of `Parser` is object. (You won't notice that unless you use `Parser` class directly)
- `SyntaxError` renamed to `SyntaxErrorException`
- `Items::__construct` accepts the options array instead of separate arguments. (You won't notice that unless you instantiate `Items` class directly)
- `Lexer` renamed to `Tokens`
- `DebugLexer` renamed to `TokensWithDebugging`
### Added
- Multiple JSON Pointers can be specified as an array in `pointer` option. See README. Thanks @fwolfsjaeger.
- New methods available during iteration: `Items::getCurrentJsonPointer()` and `Items::getMatchedJsonPointer()`
to track where you are. See README. Thanks @fwolfsjaeger.
### Fixed
- Incorrect position information of `TokensWithDebugging::getPosition()`. Was constantly off by 1-2 bytes.
## 0.8.0
### Changed
- Internal decoders moved to `ItemDecoder`. `ErrorWrappingDecoder` decorator now requires `ItemDecoder` as well.
- Dropped PHP 5.6 support.
### Deprecated
- `JsonMachine\JsonMachine` entry point class is deprecated, use `JsonMachine\Items` instead.
- `JsonMachine\JsonDecoder\Decoder` interface is deprecated. Use `JsonMachine\JsonDecoder\ItemDecoder` instead.
### Added
- New entry point class `Items` replaces `JsonMachine`.
- Object as default decoding structure instead of array in `Items`.
- `Items::getIterator()` now returns `Parser`'s iterator directly. Call `Items::getIterator()`
instead of `JsonMachine::getIterator()::getIterator()` to get to `Parser`'s iterator if you need it. Fixes
https://stackoverflow.com/questions/63706550
- `Items` uses `options` in its factory methods instead of growing number of many parameters. See **Options** in README.
- `Items` introduces new `debug` option. See **Options** in README.
- Noticeable performance improvements. What took 10 seconds in `0.7.*` takes **about** 7 seconds in `0.8.0`.
## 0.7.1
### New features
- PHP 8.1 support
- DEV: Build system switched to composer scripts and Makefile
## 0.7.0
### New features
- Use a `-` in json pointer as a wildcard for an array index. Example: `/users/-/id`. Thanks @cerbero90
## 0.6.1
### Fixed bugs
- Empty dict at the end of an item was causing Syntax error in the next item. Reason: closing `}` did not set object key expectation to `false`. (#41 via PR #42).
## 0.6.0
### New features
- **New:** Json pointer can find scalar values in JSON document as well as iterable values. See
[Getting single scalar values](README.md#getting-scalar-values)
- Parser ends when the end of the desired data is reached and does not heat up the atmosphere further.
- Optimizations: about 15% speed gain.
### BC breaks
- A json pointer that matches scalar value does not throw anymore, but the scalar value is yielded in foreach.
## 0.5.0
### New features
- Introduced `FileChunks` class. Takes care of the proper resource management when iterating via `JsonMachine::fromFile()`.
It is used internally, and you probably won't come across it.
- New `ErrorWrappingDecoder`. Use it when you want to skip malformed JSON items. See [Decoders](README.md#decoders).
### BC breaks
- `StreamBytes` and `StringBytes` renamed to `StreamChunks` and `StringChunks`.
These are internal classes, and you probably won't notice the change
unless you use them directly for some reason.
## 0.4.1
### New features
- Tracking of parsing progress
## 0.4.0
### New features
- [Decoders](README.md#decoders)
- PHP 8 support (thanks @snapshotpl)
### BC breaks
- `ext-json` is not required in `composer.json` anymore, because custom decoder might not need it.
However **built-in decoders depend on it** so it must be present if you use them.
- All exceptions now extend `JsonMachineException` (thanks @gabimem)
- Throws `UnexpectedEndSyntaxErrorException` on an unexpected end of JSON structure (thanks @gabimem)
- Function `httpClientChunks()` is **deprecated** so that compatibility with Symfony HttpClient
is not on the shoulders of JSON Machine maintainer. The code is simple and everyone can make their own
function and maintain it. The code was moved to [examples](examples/symfonyHttpClient.php).
- Function `objects()` is **deprecated**. The way `objects()` works is that it casts decoded arrays
to objects. It brings some unnecessary overhead and risks on huge datasets.
Alternative is to use `ExtJsonDecoder` which decodes items as objects by default (same as `json_decode`).
```php
chunks = $bytesIterator;
$this->jsonPointer = $options['pointer'];
$this->jsonDecoder = $options['decoder'];
$this->debugEnabled = $options['debug'];
if ($this->debugEnabled) {
$tokensClass = TokensWithDebugging::class;
} else {
$tokensClass = Tokens::class;
}
$this->parser = new Parser(
new $tokensClass(
$this->chunks
),
$this->jsonPointer,
$this->jsonDecoder ?: new ExtJsonDecoder()
);
}
/**
* @param string $string
*
* @return self
*
* @throws InvalidArgumentException
*/
public static function fromString($string, array $options = [])
{
return new self(new StringChunks($string), $options);
}
/**
* @param string $file
*
* @return self
*
* @throws Exception\InvalidArgumentException
*/
public static function fromFile($file, array $options = [])
{
return new self(new FileChunks($file), $options);
}
/**
* @param resource $stream
*
* @return self
*
* @throws Exception\InvalidArgumentException
*/
public static function fromStream($stream, array $options = [])
{
return new self(new StreamChunks($stream), $options);
}
/**
* @param iterable $iterable
*
* @return self
*
* @throws Exception\InvalidArgumentException
*/
public static function fromIterable($iterable, array $options = [])
{
return new self($iterable, $options);
}
#[\ReturnTypeWillChange]
public function getIterator()
{
return $this->parser->getIterator();
}
public function getPosition()
{
return $this->parser->getPosition();
}
public function getJsonPointers(): array
{
return $this->parser->getJsonPointers();
}
public function getCurrentJsonPointer(): string
{
return $this->parser->getCurrentJsonPointer();
}
public function getMatchedJsonPointer(): string
{
return $this->parser->getMatchedJsonPointer();
}
/**
* @return bool
*/
public function isDebugEnabled()
{
return $this->debugEnabled;
}
}
PK ! Tr r src/ItemsOptions.phpnu [ validateOptions($options);
parent::__construct($this->options);
}
public function toArray(): array
{
return $this->options;
}
/**
* @throws InvalidArgumentException
*/
private function validateOptions(array $options)
{
$mergedOptions = array_merge(self::defaultOptions(), $options);
try {
foreach ($mergedOptions as $optionName => $optionValue) {
if ( ! isset(self::defaultOptions()[$optionName])) {
throw new InvalidArgumentException("Option '$optionName' does not exist.");
}
$this->options[$optionName] = $this->{"opt_$optionName"}($optionValue);
}
} catch (\TypeError $typeError) {
throw new InvalidArgumentException(
preg_replace('~Argument #[0-9]+~', "Option '$optionName'", $typeError->getMessage())
);
}
}
private function opt_pointer($pointer)
{
if (is_array($pointer)) {
(function (string ...$p) {})(...$pointer);
} else {
(function (string $p) {})($pointer);
}
return $pointer;
}
private function opt_decoder(ItemDecoder $decoder = null)
{
return $decoder;
}
private function opt_debug(bool $debug)
{
return $debug;
}
public static function defaultOptions(): array
{
return [
'pointer' => '',
'decoder' => new ExtJsonDecoder(),
'debug' => false,
];
}
}
PK ! Kj src/FileChunks.phpnu [ fileName = $fileName;
$this->chunkSize = $chunkSize;
}
/**
* @return \Generator
*/
#[\ReturnTypeWillChange]
public function getIterator()
{
$fileHandle = fopen($this->fileName, 'r');
try {
yield from new StreamChunks($fileHandle, $this->chunkSize);
} finally {
fclose($fileHandle);
}
}
}
PK ! src/PositionAware.phpnu [ stream = $stream;
$this->chunkSize = $chunkSize;
}
/**
* @return \Generator
*/
#[\ReturnTypeWillChange]
public function getIterator()
{
while ('' !== ($chunk = fread($this->stream, $this->chunkSize))) {
yield $chunk;
}
}
}
PK ! L$u u & src/Exception/JsonMachineException.phpnu [ malformedJson = $malformedJson;
$this->errorMessage = $errorMessage;
}
/**
* @return string
*/
public function getMalformedJson()
{
return $this->malformedJson;
}
/**
* @return string
*/
public function getErrorMessage()
{
return $this->errorMessage;
}
}
PK ! d src/JsonDecoder/ItemDecoder.phpnu [ assoc = $assoc;
$this->depth = $depth;
$this->options = $options;
}
public function decode($jsonValue)
{
$decoded = json_decode($jsonValue, $this->assoc, $this->depth, $this->options);
if (json_last_error() !== JSON_ERROR_NONE) {
return new InvalidResult(json_last_error_msg());
}
return new ValidResult($decoded);
}
}
PK ! !l ! src/JsonDecoder/InvalidResult.phpnu [ errorMessage = $errorMessage;
}
public function getErrorMessage(): string
{
return $this->errorMessage;
}
public function isOk(): bool
{
return false;
}
}
PK ! 쩭 # src/JsonDecoder/PassThruDecoder.phpnu [ innerDecoder = $innerDecoder;
}
public function decode($jsonValue)
{
$result = $this->innerDecoder->decode($jsonValue);
if ( ! $result->isOk()) {
return new ValidResult(new DecodingError($jsonValue, $result->getErrorMessage()));
}
return $result;
}
}
PK ! 8u src/JsonDecoder/ValidResult.phpnu [ value = $value;
}
/**
* @return mixed
*/
public function getValue()
{
return $this->value;
}
public function isOk(): bool
{
return true;
}
}
PK ! * L L src/Tokens.phpnu [ $jsonChunks
*/
public function __construct($jsonChunks)
{
$this->jsonChunks = $jsonChunks;
}
/**
* @return Generator
*/
#[\ReturnTypeWillChange]
public function getIterator()
{
$insignificantBytes = $this->insignificantBytes();
$tokenBoundaries = $this->tokenBoundaries();
$colonCommaBracket = $this->colonCommaBracketTokenBoundaries();
$inString = false;
$tokenBuffer = '';
$escaping = false;
foreach ($this->jsonChunks as $jsonChunk) {
$bytesLength = strlen($jsonChunk);
for ($i = 0; $i < $bytesLength; ++$i) {
$byte = $jsonChunk[$i];
if ($escaping) {
$escaping = false;
$tokenBuffer .= $byte;
continue;
}
if (isset($insignificantBytes[$byte])) { // is a JSON-structure insignificant byte
$tokenBuffer .= $byte;
continue;
}
if ($inString) {
if ($byte == '"') {
$inString = false;
} elseif ($byte == '\\') {
$escaping = true;
}
$tokenBuffer .= $byte;
continue;
}
if (isset($tokenBoundaries[$byte])) {
if ($tokenBuffer != '') {
yield $tokenBuffer;
$tokenBuffer = '';
}
if (isset($colonCommaBracket[$byte])) {
yield $byte;
}
} else { // else branch matches `"` but also `\` outside of a string literal which is an error anyway but strictly speaking not correctly parsed token
$inString = true;
$tokenBuffer .= $byte;
}
}
}
if ($tokenBuffer != '') {
yield $tokenBuffer;
}
}
private function tokenBoundaries()
{
$utf8bom1 = "\xEF";
$utf8bom2 = "\xBB";
$utf8bom3 = "\xBF";
return array_merge(
[
$utf8bom1 => true,
$utf8bom2 => true,
$utf8bom3 => true,
' ' => true,
"\n" => true,
"\r" => true,
"\t" => true,
],
$this->colonCommaBracketTokenBoundaries()
);
}
private function colonCommaBracketTokenBoundaries(): array
{
return [
'{' => true,
'}' => true,
'[' => true,
']' => true,
':' => true,
',' => true,
];
}
private function insignificantBytes(): array
{
$insignificantBytes = [];
foreach (range(0, 255) as $ord) {
if ( ! in_array(
chr($ord),
['\\', '"', "\xEF", "\xBB", "\xBF", ' ', "\n", "\r", "\t", '{', '}', '[', ']', ':', ',']
)) {
$insignificantBytes[chr($ord)] = true;
}
}
return $insignificantBytes;
}
public function getPosition(): int
{
return 0;
}
public function getLine(): int
{
return 1;
}
public function getColumn(): int
{
return 0;
}
}
PK ! eT$9 $9 src/Parser.phpnu [ toArray();
$this->tokens = $tokens;
$this->jsonDecoder = $jsonDecoder ?: new ExtJsonDecoder();
$this->hasSingleJsonPointer = (count($jsonPointers) === 1);
$this->jsonPointers = array_combine($jsonPointers, $jsonPointers);
$this->paths = $this->buildPaths($this->jsonPointers);
}
private function buildPaths(array $jsonPointers): array
{
return array_map(function ($jsonPointer) {
return self::jsonPointerToPath($jsonPointer);
}, $jsonPointers);
}
/**
* @return \Generator
*
* @throws PathNotFoundException
*/
#[\ReturnTypeWillChange]
public function getIterator()
{
$tokenTypes = $this->tokenTypes();
$iteratorStruct = null;
$currentPath = &$this->currentPath;
$currentPath = [];
$currentPathWildcard = [];
$pointersFound = [];
$currentLevel = -1;
$stack = [$currentLevel => null];
$jsonBuffer = '';
$key = null;
$objectKeyExpected = false;
$inObject = true; // hack to make "!$inObject" in first iteration work. Better code structure?
$expectedType = self::OBJECT_START | self::ARRAY_START;
$subtreeEnded = false;
$token = null;
$currentPathChanged = true;
$jsonPointerPath = [];
$iteratorLevel = 0;
// local variables for faster name lookups
$tokens = $this->tokens;
foreach ($tokens as $token) {
if ($currentPathChanged) {
$currentPathChanged = false;
$jsonPointerPath = $this->getMatchingJsonPointerPath();
$iteratorLevel = count($jsonPointerPath);
}
$tokenType = $tokenTypes[$token[0]];
if (0 == ($tokenType & $expectedType)) {
$this->error('Unexpected symbol', $token);
}
$isValue = ($tokenType | 23) == 23; // 23 = self::ANY_VALUE
if ( ! $inObject && $isValue && $currentLevel < $iteratorLevel) {
$currentPathChanged = ! $this->hasSingleJsonPointer;
$currentPath[$currentLevel] = isset($currentPath[$currentLevel]) ? (string) (1 + (int) $currentPath[$currentLevel]) : '0';
$currentPathWildcard[$currentLevel] = preg_match('/^(?:\d+|-)$/S', $jsonPointerPath[$currentLevel]) ? '-' : $currentPath[$currentLevel];
unset($currentPath[$currentLevel + 1], $currentPathWildcard[$currentLevel + 1], $stack[$currentLevel + 1]);
}
if (
(
$jsonPointerPath == $currentPath
|| $jsonPointerPath == $currentPathWildcard
)
&& (
$currentLevel > $iteratorLevel
|| (
! $objectKeyExpected
&& (
($currentLevel == $iteratorLevel && $isValue)
|| ($currentLevel + 1 == $iteratorLevel && ($tokenType | 3) == 3) // 3 = self::SCALAR_VALUE
)
)
)
) {
$jsonBuffer .= $token;
}
// todo move this switch to the top just after the syntax check to be a correct FSM
switch ($token[0]) {
case '"':
if ($objectKeyExpected) {
$objectKeyExpected = false;
$expectedType = 128; // 128 = self::COLON
if ($currentLevel == $iteratorLevel) {
$key = $token;
} elseif ($currentLevel < $iteratorLevel) {
$key = $token;
$referenceToken = substr($token, 1, -1);
$currentPathChanged = ! $this->hasSingleJsonPointer;
$currentPath[$currentLevel] = $referenceToken;
$currentPathWildcard[$currentLevel] = $referenceToken;
unset($currentPath[$currentLevel + 1], $currentPathWildcard[$currentLevel + 1]);
}
continue 2; // a valid json chunk is not completed yet
}
if ($inObject) {
$expectedType = 72; // 72 = self::AFTER_OBJECT_VALUE;
} else {
$expectedType = 96; // 96 = self::AFTER_ARRAY_VALUE;
}
break;
case ',':
if ($inObject) {
$objectKeyExpected = true;
$expectedType = 2; // 2 = self::SCALAR_STRING
} else {
$expectedType = 23; // 23 = self::ANY_VALUE
}
continue 2; // a valid json chunk is not completed yet
case ':':
$expectedType = 23; // 23 = self::ANY_VALUE
continue 2; // a valid json chunk is not completed yet
case '{':
++$currentLevel;
if ($currentLevel <= $iteratorLevel) {
$iteratorStruct = '{';
}
$stack[$currentLevel] = '{';
$inObject = true;
$expectedType = 10; // 10 = self::AFTER_OBJECT_START
$objectKeyExpected = true;
continue 2; // a valid json chunk is not completed yet
case '[':
++$currentLevel;
if ($currentLevel <= $iteratorLevel) {
$iteratorStruct = '[';
}
$stack[$currentLevel] = '[';
$inObject = false;
$expectedType = 55; // 55 = self::AFTER_ARRAY_START;
continue 2; // a valid json chunk is not completed yet
case '}':
$objectKeyExpected = false;
// no break
case ']':
--$currentLevel;
$inObject = $stack[$currentLevel] == '{';
// no break
default:
if ($inObject) {
$expectedType = 72; // 72 = self::AFTER_OBJECT_VALUE;
} else {
$expectedType = 96; // 96 = self::AFTER_ARRAY_VALUE;
}
}
if ($currentLevel > $iteratorLevel) {
continue; // a valid json chunk is not completed yet
}
if ($jsonBuffer !== '') {
$valueResult = $this->jsonDecoder->decode($jsonBuffer);
$jsonBuffer = '';
if ( ! $valueResult->isOk()) {
$this->error($valueResult->getErrorMessage(), $token);
}
if ($iteratorStruct == '[') {
yield $valueResult->getValue();
} else {
$keyResult = $this->jsonDecoder->decode($key);
if ( ! $keyResult->isOk()) {
$this->error($keyResult->getErrorMessage(), $key);
}
yield $keyResult->getValue() => $valueResult->getValue();
unset($keyResult);
}
unset($valueResult);
}
if ($jsonPointerPath == $currentPath || $jsonPointerPath == $currentPathWildcard) {
if ( ! in_array($this->matchedJsonPointer, $pointersFound, true)) {
$pointersFound[] = $this->matchedJsonPointer;
}
} elseif (count($pointersFound) == count($this->jsonPointers)) {
$subtreeEnded = true;
break;
}
}
if ($token === null) {
$this->error('Cannot iterate empty JSON', $token);
}
if ($currentLevel > -1 && ! $subtreeEnded) {
$this->error('JSON string ended unexpectedly', $token, UnexpectedEndSyntaxErrorException::class);
}
if (count($pointersFound) !== count($this->jsonPointers)) {
throw new PathNotFoundException(sprintf("Paths '%s' were not found in json stream.", implode(', ', array_diff($this->jsonPointers, $pointersFound))));
}
$this->matchedJsonPointer = null;
$this->currentPath = null;
}
private function tokenTypes()
{
return [
'n' => self::SCALAR_CONST,
't' => self::SCALAR_CONST,
'f' => self::SCALAR_CONST,
'-' => self::SCALAR_CONST,
'0' => self::SCALAR_CONST,
'1' => self::SCALAR_CONST,
'2' => self::SCALAR_CONST,
'3' => self::SCALAR_CONST,
'4' => self::SCALAR_CONST,
'5' => self::SCALAR_CONST,
'6' => self::SCALAR_CONST,
'7' => self::SCALAR_CONST,
'8' => self::SCALAR_CONST,
'9' => self::SCALAR_CONST,
'"' => self::SCALAR_STRING,
'{' => self::OBJECT_START,
'}' => self::OBJECT_END,
'[' => self::ARRAY_START,
']' => self::ARRAY_END,
',' => self::COMMA,
':' => self::COLON,
];
}
private function getMatchingJsonPointerPath(): array
{
$matchingPointer = key($this->paths);
if (count($this->paths) === 1) {
$this->matchedJsonPointer = $matchingPointer;
return $this->paths[$matchingPointer];
}
$currentPathLength = count($this->currentPath);
$matchLength = -1;
foreach ($this->paths as $jsonPointer => $path) {
$matchingReferenceTokens = [];
foreach ($path as $i => $referenceToken) {
if (
! isset($this->currentPath[$i])
|| (
$this->currentPath[$i] !== $referenceToken
&& ValidJsonPointers::wildcardify($this->currentPath[$i]) !== $referenceToken
)
) {
continue;
}
$matchingReferenceTokens[$i] = $referenceToken;
}
if (empty($matchingReferenceTokens)) {
continue;
}
$currentMatchLength = count($matchingReferenceTokens);
if ($currentMatchLength > $matchLength) {
$matchingPointer = $jsonPointer;
$matchLength = $currentMatchLength;
}
if ($matchLength === $currentPathLength) {
break;
}
}
$this->matchedJsonPointer = $matchingPointer;
return $this->paths[$matchingPointer];
}
public function getJsonPointers(): array
{
return array_values($this->jsonPointers);
}
public function getCurrentJsonPointer(): string
{
if ($this->currentPath === null) {
throw new JsonMachineException(__METHOD__.' must be called inside a loop');
}
return self::pathToJsonPointer($this->currentPath);
}
public function getMatchedJsonPointer(): string
{
if ($this->matchedJsonPointer === null) {
throw new JsonMachineException(__METHOD__.' must be called inside a loop');
}
return $this->matchedJsonPointer;
}
/**
* @param string $msg
* @param string $token
* @param string $exception
*/
private function error($msg, $token, $exception = SyntaxErrorException::class)
{
throw new $exception($msg." '".$token."'", $this->tokens->getPosition());
}
/**
* @return int
*
* @throws JsonMachineException
*/
public function getPosition()
{
if ($this->tokens instanceof PositionAware) {
return $this->tokens->getPosition();
}
throw new JsonMachineException('Provided tokens iterable must implement PositionAware to call getPosition on it.');
}
private static function jsonPointerToPath(string $jsonPointer): array
{
return array_slice(array_map(function ($jsonPointerPart) {
return str_replace(['~1', '~0'], ['/', '~'], $jsonPointerPart);
}, explode('/', $jsonPointer)), 1);
}
private static function pathToJsonPointer(array $path): string
{
$encodedParts = array_map(function ($addressPart) {
return str_replace(['~', '/'], ['~0', '~1'], $addressPart);
}, $path);
array_unshift($encodedParts, '');
return implode('/', $encodedParts);
}
}
PK ! " src/StringChunks.phpnu [ string = $string;
$this->chunkSize = $chunkSize;
}
/**
* @return \Generator
*/
#[\ReturnTypeWillChange]
public function getIterator()
{
$len = strlen($this->string);
for ($i = 0; $i < $len; $i += $this->chunkSize) {
yield substr($this->string, $i, $this->chunkSize);
}
}
}
PK ! # src/TokensWithDebugging.phpnu [ $jsonChunks
*/
public function __construct($jsonChunks)
{
$this->jsonChunks = $jsonChunks;
}
/**
* @return \Generator
*/
#[\ReturnTypeWillChange]
public function getIterator()
{
// Treat UTF-8 BOM bytes as whitespace
${"\xEF"} = ${"\xBB"} = ${"\xBF"} = 0;
${' '} = 0;
${"\n"} = 0;
${"\r"} = 0;
${"\t"} = 0;
${'{'} = 1;
${'}'} = 1;
${'['} = 1;
${']'} = 1;
${':'} = 1;
${','} = 1;
$inString = false;
$tokenBuffer = '';
$escaping = false;
$tokenWidth = 0;
$ignoreLF = false;
$position = 0;
$line = 1;
$column = 0;
foreach ($this->jsonChunks as $bytes) {
$bytesLength = strlen($bytes);
for ($i = 0; $i < $bytesLength; ++$i) {
$byte = $bytes[$i];
if ($inString) {
if ($byte == '"' && ! $escaping) {
$inString = false;
}
$escaping = ($byte == '\\' && ! $escaping);
$tokenBuffer .= $byte;
++$tokenWidth;
continue;
}
if (isset($$byte)) {
++$column;
if ($tokenBuffer != '') {
$this->position = $position + $i;
$this->column = $column;
$this->line = $line;
yield $tokenBuffer;
$column += $tokenWidth;
$tokenBuffer = '';
$tokenWidth = 0;
}
if ($$byte) { // is not whitespace
$this->position = $position + $i + 1;
$this->column = $column;
$this->line = $line;
yield $byte;
// track line number and reset column for each newline
} elseif ($byte == "\n") {
// handle CRLF newlines
if ($ignoreLF) {
--$column;
$ignoreLF = false;
continue;
}
++$line;
$column = 0;
} elseif ($byte == "\r") {
++$line;
$ignoreLF = true;
$column = 0;
}
} else {
if ($byte == '"') {
$inString = true;
}
$tokenBuffer .= $byte;
++$tokenWidth;
}
}
$position += $i;
}
$this->position = $position;
if ($tokenBuffer != '') {
$this->column = $column;
yield $tokenBuffer;
}
}
/**
* @return int
*/
public function getPosition()
{
return $this->position;
}
/**
* Returns the line number of the lexeme currently being processed (index starts at one).
*
* @return int
*/
public function getLine()
{
return $this->line;
}
/**
* The position of currently being processed lexeme within the line (index starts at one).
*
* @return int
*/
public function getColumn()
{
return $this->column;
}
}
PK ! <
src/ValidJsonPointers.phpnu [ jsonPointers = array_values($jsonPointers);
}
/**
* @throws InvalidArgumentException
*/
public function toArray(): array
{
if ( ! $this->validated) {
$this->validate();
}
return $this->jsonPointers;
}
/**
* @throws InvalidArgumentException
*/
private function validate()
{
$this->validateFormat();
$this->validateJsonPointersDoNotIntersect();
$this->validated = true;
}
/**
* @throws InvalidArgumentException
*/
private function validateFormat()
{
foreach ($this->jsonPointers as $jsonPointerEl) {
if (preg_match('_^(/(([^/~])|(~[01]))*)*$_', $jsonPointerEl) === 0) {
throw new InvalidArgumentException(
sprintf("Given value '%s' of \$jsonPointer is not valid JSON Pointer", $jsonPointerEl)
);
}
}
}
/**
* @throws InvalidArgumentException
*/
private function validateJsonPointersDoNotIntersect()
{
foreach ($this->jsonPointers as $keyA => $jsonPointerA) {
foreach ($this->jsonPointers as $keyB => $jsonPointerB) {
if ($keyA === $keyB) {
continue;
}
if ($jsonPointerA === $jsonPointerB
|| self::str_contains($jsonPointerA, $jsonPointerB)
|| self::str_contains($jsonPointerA, self::wildcardify($jsonPointerB))
) {
throw new InvalidArgumentException(
sprintf(
"JSON Pointers must not intersect. At least these two do: '%s', '%s'",
$jsonPointerA,
$jsonPointerB
)
);
}
}
}
}
public static function wildcardify(string $jsonPointerPart): string
{
return preg_replace('~/\d+(/|$)~S', '/-$1', $jsonPointerPart);
}
/**
* @see https://github.com/symfony/polyfill/blob/v1.24.0/src/Php80/Php80.php
*/
public static function str_contains(string $haystack, string $needle): bool
{
return '' === $needle || false !== strpos($haystack, $needle);
}
}
PK ! [yCR, R, LICENSE.txtnu [ Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [2018] [Filip Halaxa]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
PK ! MD{Z Z README.mdnu [ 
(README in sync with the code)
Very easy to use and memory efficient drop-in replacement for inefficient iteration of big JSON files or streams
for PHP >=7.0. See [TL;DR](#tl-dr). No dependencies in production except optional `ext-json`.
[](https://github.com/halaxa/json-machine/actions)
[](https://codecov.io/gh/halaxa/json-machine)
[](https://packagist.org/packages/halaxa/json-machine)
[](https://packagist.org/packages/halaxa/json-machine)
---
* [TL;DR](#tl-dr)
* [Introduction](#introduction)
* [Parsing JSON documents](#parsing-json-documents)
+ [Parsing a document](#simple-document)
+ [Parsing a subtree](#parsing-a-subtree)
+ [Parsing nested values in arrays](#parsing-nested-values)
+ [Parsing a single scalar value](#getting-scalar-values)
+ [Parsing multiple subtrees](#parsing-multiple-subtrees)
+ [What is JSON Pointer anyway?](#json-pointer)
* [Options](#options)
* [Parsing streaming responses from a JSON API](#parsing-json-stream-api-responses)
+ [GuzzleHttp](#guzzlehttp)
+ [Symfony HttpClient](#symfony-httpclient)
* [Tracking the progress](#tracking-parsing-progress)
* [Decoders](#decoders)
+ [Available decoders](#available-decoders)
* [Error handling](#error-handling)
+ [Catching malformed items](#malformed-items)
* [Parser efficiency](#on-parser-efficiency)
+ [Streams / files](#streams-files)
+ [In-memory JSON strings](#in-memory-json-strings)
* [Troubleshooting](#troubleshooting)
+ ["I'm still getting Allowed memory size ... exhausted"](#step1)
+ ["That didn't help"](#step2)
+ ["I am still out of luck"](#step3)
* [Installation](#installation)
* [Development](#development)
+ [Non containerized](#non-containerized)
+ [Containerized](#containerized)
* [Support](#support)
* [License](#license)
---
## TL;DR
```diff
$user) {
// just process $user as usual
var_dump($user->name);
}
```
Random access like `$users[42]` is not yet possible.
Use above-mentioned `foreach` and find the item or use [JSON Pointer](#parsing-a-subtree).
Count the items via [`iterator_count($users)`](https://www.php.net/manual/en/function.iterator-count.php).
Remember it will still have to internally iterate the whole thing to get the count and thus will take about the same time.
Requires `ext-json` if used out of the box. See [Decoders](#decoders).
Follow [CHANGELOG](CHANGELOG.md).
## Introduction
JSON Machine is an efficient, easy-to-use and fast JSON stream/pull/incremental/lazy (whatever you name it) parser
based on generators developed for unpredictably long JSON streams or documents. Main features are:
- Constant memory footprint for unpredictably large JSON documents.
- Ease of use. Just iterate JSON of any size with `foreach`. No events and callbacks.
- Efficient iteration on any subtree of the document, specified by [JSON Pointer](#json-pointer)
- Speed. Performance critical code contains no unnecessary function calls, no regular expressions
and uses native `json_decode` to decode JSON document items by default. See [Decoders](#decoders).
- Parses not only streams but any iterable that produces JSON chunks.
- Thoroughly tested. More than 200 tests and 1000 assertions.
## Parsing JSON documents
### Parsing a document
Let's say that `fruits.json` contains this huge JSON document:
```json
// fruits.json
{
"apple": {
"color": "red"
},
"pear": {
"color": "yellow"
}
}
```
It can be parsed this way:
```php
$data) {
// 1st iteration: $name === "apple" and $data->color === "red"
// 2nd iteration: $name === "pear" and $data->color === "yellow"
}
```
Parsing a json array instead of a json object follows the same logic.
The key in a foreach will be a numeric index of an item.
If you prefer JSON Machine to return arrays instead of objects, use `new ExtJsonDecoder(true)` as a decoder.
```php
new ExtJsonDecoder(true)]);
```
### Parsing a subtree
If you want to iterate only `results` subtree in this `fruits.json`:
```json
// fruits.json
{
"results": {
"apple": {
"color": "red"
},
"pear": {
"color": "yellow"
}
}
}
```
use JSON Pointer `/results` as `pointer` option:
```php
'/results']);
foreach ($fruits as $name => $data) {
// The same as above, which means:
// 1st iteration: $name === "apple" and $data->color === "red"
// 2nd iteration: $name === "pear" and $data->color === "yellow"
}
```
> Note:
>
> Value of `results` is not loaded into memory at once, but only one item in
> `results` at a time. It is always one item in memory at a time at the level/subtree
> you are currently iterating. Thus, the memory consumption is constant.
### Parsing nested values in arrays
The JSON Pointer spec also allows to use a hyphen (`-`) instead of a specific array index. JSON Machine interprets
it as a wildcard which matches any **array index** (not any object key). This enables you to iterate nested values in
arrays without loading the whole item.
Example:
```json
// fruitsArray.json
{
"results": [
{
"name": "apple",
"color": "red"
},
{
"name": "pear",
"color": "yellow"
}
]
}
```
To iterate over all colors of the fruits, use the JSON Pointer `"/results/-/color"`.
```php
'/results/-/color']);
foreach ($fruits as $key => $value) {
// 1st iteration:
$key == 'color';
$value == 'red';
$fruits->getMatchedJsonPointer() == '/results/-/color';
$fruits->getCurrentJsonPointer() == '/results/0/color';
// 2nd iteration:
$key == 'color';
$value == 'yellow';
$fruits->getMatchedJsonPointer() == '/results/-/color';
$fruits->getCurrentJsonPointer() == '/results/1/color';
}
```
### Parsing a single scalar value
You can parse a single scalar value anywhere in the document the same way as a collection. Consider this example:
```json
// fruits.json
{
"lastModified": "2012-12-12",
"apple": {
"color": "red"
},
"pear": {
"color": "yellow"
},
// ... gigabytes follow ...
}
```
Get the scalar value of `lastModified` key like this:
```php
'/lastModified']);
foreach ($fruits as $key => $value) {
// 1st and final iteration:
// $key === 'lastModified'
// $value === '2012-12-12'
}
```
When parser finds the value and yields it to you, it stops parsing. So when a single scalar value is in the beginning
of a gigabytes-sized file or stream, it just gets the value from the beginning in no time and with almost no memory
consumed.
The obvious shortcut is:
```php
'/lastModified']);
$lastModified = iterator_to_array($fruits)['lastModified'];
```
Single scalar value access supports array indices in JSON Pointer as well.
### Parsing multiple subtrees
It is also possible to parse multiple subtrees using multiple JSON Pointers. Consider this example:
```json
// fruits.json
{
"lastModified": "2012-12-12",
"berries": [
{
"name": "strawberry", // not a berry, but whatever ...
"color": "red"
},
{
"name": "raspberry", // the same ...
"color": "red"
}
],
"citruses": [
{
"name": "orange",
"color": "orange"
},
{
"name": "lime",
"color": "green"
}
]
}
```
To iterate over all berries and citrus fruits, use the JSON pointers `["/berries", "/citrus"]`. The order of pointers
does not matter. The items will be iterated in the order of appearance in the document.
```php
['/berries', '/citruses']
]);
foreach ($fruits as $key => $value) {
// 1st iteration:
$value == ["name" => "strawberry", "color" => "red"];
$fruits->getCurrentJsonPointer() == '/berries';
// 2nd iteration:
$value == ["name" => "raspberry", "color" => "red"];
$fruits->getCurrentJsonPointer() == '/berries';
// 3rd iteration:
$value == ["name" => "orange", "color" => "orange"];
$fruits->getCurrentJsonPointer() == '/citruses';
// 4th iteration:
$value == ["name" => "lime", "color" => "green"];
$fruits->getCurrentJsonPointer() == '/citruses';
}
```
### What is JSON Pointer anyway?
It's a way of addressing one item in JSON document. See the [JSON Pointer RFC 6901](https://tools.ietf.org/html/rfc6901).
It's very handy, because sometimes the JSON structure goes deeper, and you want to iterate a subtree,
not the main level. So you just specify the pointer to the JSON array or object (or even to a scalar value) you want to iterate and off you go.
When the parser hits the collection you specified, iteration begins. You can pass it as `pointer` option in all
`Items::from*` functions. If you specify a pointer to a non-existent position in the document, an exception is thrown.
It can be used to access scalar values as well. **JSON Pointer itself must be a valid JSON string**. Literal comparison
of reference tokens (the parts between slashes) is performed against the JSON document keys/member names.
Some examples:
| JSON Pointer value | Will iterate through |
|--------------------------|-----------------------------------------------------------------------------------------------------------|
| (empty string - default) | `["this", "array"]` or `{"a": "this", "b": "object"}` will be iterated (main level) |
| `/result/items` | `{"result": {"items": ["this", "array", "will", "be", "iterated"]}}` |
| `/0/items` | `[{"items": ["this", "array", "will", "be", "iterated"]}]` (supports array indices) |
| `/results/-/status` | `{"results": [{"status": "iterated"}, {"status": "also iterated"}]}` (a hyphen as an array index wildcard)|
| `/` (gotcha! - a slash followed by an empty string, see the [spec](https://tools.ietf.org/html/rfc6901#section-5)) | `{"":["this","array","will","be","iterated"]}` |
| `/quotes\"` | `{"quotes\"": ["this", "array", "will", "be", "iterated"]}` |
## Options
Options may change how a JSON is parsed. Array of options is the second parameter of all `Items::from*` functions.
Available options are:
- `pointer` - A JSON Pointer string that tells which part of the document you want to iterate.
- `decoder` - An instance of `ItemDecoder` interface.
- `debug` - `true` or `false` to enable or disable the debug mode. When the debug mode is enabled, data such as line,
column and position in the document are available during parsing or in exceptions. Keeping debug disabled adds slight
performance advantage.
## Parsing streaming responses from a JSON API
A stream API response or any other JSON stream is parsed exactly the same way as file is. The only difference
is, you use `Items::fromStream($streamResource)` for it, where `$streamResource` is the stream
resource with the JSON document. The rest is the same as with parsing files. Here are some examples of
popular http clients which support streaming responses:
### GuzzleHttp
Guzzle uses its own streams, but they can be converted back to PHP streams by calling
`\GuzzleHttp\Psr7\StreamWrapper::getResource()`. Pass the result of this function to
`Items::fromStream` function, and you're set up. See working
[GuzzleHttp example](examples/guzzleHttp.php).
### Symfony HttpClient
A stream response of Symfony HttpClient works as iterator. And because JSON Machine is
based on iterators, the integration with Symfony HttpClient is very simple. See
[HttpClient example](examples/symfonyHttpClient.php).
## Tracking the progress (with `debug` enabled)
Big documents may take a while to parse. Call `Items::getPosition()` in your `foreach` to get current
count of the processed bytes from the beginning. Percentage is then easy to calculate as `position / total * 100`.
To find out the total size of your document in bytes you may want to check:
- `strlen($document)` if you parse a string
- `filesize($file)` if you parse a file
- `Content-Length` http header if you parse a http stream response
- ... you get the point
If `debug` is disabled, `getPosition()` always returns `0`.
```php
true]);
foreach ($fruits as $name => $data) {
echo 'Progress: ' . intval($fruits->getPosition() / $fileSize * 100) . ' %';
}
```
## Decoders
`Items::from*` functions also accept `decoder` option. It must be an instance of
`JsonMachine\JsonDecoder\ItemDecoder`. If none is specified, `ExtJsonDecoder` is used by
default. It requires `ext-json` PHP extension to be present, because it uses
`json_decode`. When `json_decode` doesn't do what you want, implement `JsonMachine\JsonDecoder\ItemDecoder`
and make your own.
### Available decoders
- **`ExtJsonDecoder`** - **Default.** Uses `json_decode` to decode keys and values.
Constructor has the same parameters as `json_decode`.
- **`PassThruDecoder`** - Does no decoding. Both keys and values are produced as pure JSON strings.
Useful when you want to parse a JSON item with something else directly in the foreach
and don't want to implement `JsonMachine\JsonDecoder\ItemDecoder`. Since `1.0.0` does not use `json_decode`.
Example:
```php
new PassThruDecoder]);
```
- **`ErrorWrappingDecoder`** - A decorator which wraps decoding errors inside `DecodingError` object
thus enabling you to skip malformed items instead of dying on `SyntaxError` exception.
Example:
```php
new ErrorWrappingDecoder(new ExtJsonDecoder())]);
foreach ($items as $key => $item) {
if ($key instanceof DecodingError || $item instanceof DecodingError) {
// handle error of this malformed json item
continue;
}
var_dump($key, $item);
}
```
## Error handling
Since 0.4.0 every exception extends `JsonMachineException`, so you can catch that to filter any error from JSON Machine library.
### Skipping malformed items
If there's an error anywhere in a json stream, `SyntaxError` exception is thrown. That's very inconvenient,
because if there is an error inside one json item you are unable to parse the rest of the document
because of one malformed item. `ErrorWrappingDecoder` is a decoder decorator which can help you with that.
Wrap a decoder with it, and all malformed items you are iterating will be given to you in the foreach via
`DecodingError`. This way you can skip them and continue further with the document. See example in
[Available decoders](#available-decoders). Syntax errors in the structure of a json stream between the iterated
items will still throw `SyntaxError` exception though.
## Parser efficiency
The time complexity is always `O(n)`
### Streams / files
TL;DR: The memory complexity is `O(2)`
JSON Machine reads a stream (or a file) 1 JSON item at a time and generates corresponding 1 PHP item at a time.
This is the most efficient way, because if you had say 10,000 users in JSON file and wanted to parse it using
`json_decode(file_get_contents('big.json'))`, you'd have the whole string in memory as well as all the 10,000
PHP structures. Following table shows the difference:
| | String items in memory at a time | Decoded PHP items in memory at a time | Total |
|------------------------|---------------------------------:|--------------------------------------:|------:|
| `json_decode()` | 10000 | 10000 | 20000 |
| `Items::from*()` | 1 | 1 | 2 |
This means, that JSON Machine is constantly efficient for any size of processed JSON. 100 GB no problem.
### In-memory JSON strings
TL;DR: The memory complexity is `O(n+1)`
There is also a method `Items::fromString()`. If you are
forced to parse a big string, and the stream is not available, JSON Machine may be better than `json_decode`.
The reason is that unlike `json_decode`, JSON Machine still traverses the JSON string one item at a time and doesn't
load all resulting PHP structures into memory at once.
Let's continue with the example with 10,000 users. This time they are all in string in memory.
When decoding that string with `json_decode`, 10,000 arrays (objects) is created in memory and then the result
is returned. JSON Machine on the other hand creates single structure for each found item in the string and yields it back
to you. When you process this item and iterate to the next one, another single structure is created. This is the same
behaviour as with streams/files. Following table puts the concept into perspective:
| | String items in memory at a time | Decoded PHP items in memory at a time | Total |
|-----------------------------|---------------------------------:|--------------------------------------:|------:|
| `json_decode()` | 10000 | 10000 | 20000 |
| `Items::fromString()` | 10000 | 1 | 10001 |
The reality is even better. `Items::fromString` consumes about **5x less memory** than `json_decode`. The reason is
that a PHP structure takes much more memory than its corresponding JSON representation.
## Troubleshooting
### "I'm still getting Allowed memory size ... exhausted"
One of the reasons may be that the items you want to iterate over are in some sub-key such as `"results"`
but you forgot to specify a JSON Pointer. See [Parsing a subtree](#parsing-a-subtree).
### "That didn't help"
The other reason may be, that one of the items you iterate is itself so huge it cannot be decoded at once.
For example, you iterate over users and one of them has thousands of "friend" objects in it.
Use `PassThruDecoder` which does not decode an item, get the json string of the user
and parse it iteratively yourself using `Items::fromString()`.
```php
new PassThruDecoder]);
foreach ($users as $user) {
foreach (Items::fromString($user, ['pointer' => "/friends"]) as $friend) {
// process friends one by one
}
}
```
### "I am still out of luck"
It probably means that the JSON string `$user` itself or one of the friends are too big and do not fit in memory.
However, you can try this approach recursively. Parse `"/friends"` with `PassThruDecoder` getting one `$friend`
json string at a time and then parse that using `Items::fromString()`... If even that does not help,
there's probably no solution yet via JSON Machine. A feature is planned which will enable you to iterate
any structure fully recursively and strings will be served as streams.
## Installation
### Using Composer
```bash
composer require halaxa/json-machine
```
### Without Composer
Clone or download this repository and add the following to your bootstrap file:
```php
spl_autoload_register(require '/path/to/json-machine/src/autoloader.php');
```
## Development
Clone this repository. This library supports two development approaches:
1. non containerized (PHP and composer already installed on your machine)
1. containerized (Docker on your machine)
### Non containerized
Run `composer run -l` in the project dir to see available dev scripts. This way you can run some steps
of the build process such as tests.
### Containerized
[Install Docker](https://docs.docker.com/install/) and run `make` in the project dir on your host machine
to see available dev tools/commands. You can run all the steps of the build process separately as well
as the whole build process at once. Make basically runs composer dev scripts inside containers in the background.
`make build`: Runs complete build. The same command is run via GitHub Actions CI.
## Support
Do you like this library? Star it, share it, show it :)
Issues and pull requests are very welcome.
[](https://ko-fi.com/G2G57KTE4)
## License
Apache 2.0
Cogwheel element: Icons made by [TutsPlus](https://www.flaticon.com/authors/tutsplus)
from [www.flaticon.com](https://www.flaticon.com/)
is licensed by [CC 3.0 BY](http://creativecommons.org/licenses/by/3.0/)
Table of contents generated with markdown-toc
PK ! NK .php-cs-fixer.dist.phpnu [ in(__DIR__)
->exclude('ext')
;
$config = new PhpCsFixer\Config();
return $config->setRules([
'@Symfony' => true,
'not_operator_with_space' => true,
'yoda_style' => false,
'single_line_throw' => false,
'unary_operator_spaces' => false,
'visibility_required' => false,
'php_unit_test_class_requires_covers' => true,
'declare_strict_types' => true,
])
->setFinder($finder)
;
PK ! .github/FUNDING.ymlnu [ ko_fi: halaxa
PK ! [n"c c .github/workflows/makefile.ymlnu [ name: "`make build`"
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: satackey/action-docker-layer-caching@v0.0.11
# Ignore the failure of a step and avoid terminating the job.
continue-on-error: true
- name: Complete Makefile build
run: make build
- name: Code coverage report
run: make tests-coverage
- name: Send coverage report to codecov
uses: codecov/codecov-action@v2
with:
files: clover.xml
verbose: true
PK ! R Makefilenu [ .DEFAULT_GOAL := help
.PHONY: *
LATEST_PHP := 8.0 3.1.1
COVERAGE_PHP := 7.4 3.1.1
define PHP_VERSIONS
"7.0 2.7.2"\
"7.1 2.9.8"\
"7.2 3.1.1"\
"7.3 3.1.1"\
"7.4 3.1.1"\
"8.0 3.1.1"\
"8.1 3.1.1"
endef
define DOCKER_RUN
./build/docker-run.sh \
$$(./build/build-image.sh $(1)) \
$$(pwd) \
"$(2)"
endef
help:
@printf "\033[33mJSON Machine's make usage:\033[0m\n make [target] [args=\"val\"...]\n\n"
@printf "\033[33mTargets:\033[0m\n"
@grep -E '^[-a-zA-Z0-9_\.\/]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " \033[32m%-15s\033[0m\t%s\n", $$1, $$2}'
build: tests-all cs-check ## Run all necessary stuff before commit.
tests: CMD=composer tests -- $(ARGS)
tests: docker-run ## Run tests on recent PHP version. Pass args to phpunit via ARGS=""
tests-coverage: CMD=composer tests-coverage -- $(ARGS)
tests-coverage: ## Runs tests and creates ./clover.xml. Pass args to phpunit via ARGS=""
@$(call DOCKER_RUN,$(COVERAGE_PHP),$(CMD))
tests-all: ## Run tests on all supported PHP versions. Pass args to phpunit via ARGS=""
@for version in $(PHP_VERSIONS); do \
set -e; \
printf "PHP %s%.s\n" $$version; \
printf "=======\n"; \
$(call DOCKER_RUN,$$version,composer tests -- --colors=always $(ARGS)); \
printf "\n\n\n"; \
done
cs-check: CMD=composer cs-check
cs-check: docker-run ## Check code style
cs-fix: CMD=composer cs-fix
cs-fix: docker-run ## Fix code style
performance-tests: CMD=composer performance-tests
performance-tests: docker-run ## Run performance tests
release: .env build
@\
branch=$$(git branch --show-current); \
\
echo "Creating release from '$$branch'"; \
git diff --quiet --exit-code && git diff --quiet --cached --exit-code \
|| { echo "There are uncommited changes. Stopping"; exit 1; }; \
\
echo "Type the release version:"; \
read version; \
\
echo "Is README updated accordingly? [ENTER to continue]"; \
read pass; \
\
echo "Updating CHANGELOG.md"; \
$(call DOCKER_RUN,$(LATEST_PHP),php build/update-changelog.php $$version CHANGELOG.md); \
\
git diff; \
echo "Commit and tag this? [ENTER to continue]"; \
read pass; \
\
set -x; \
git commit -am "Release $$version"; \
git tag -a "$$version" -m "Release $$version"; \
set +x; \
\
echo "Push? [ENTER to continue]"; \
read pass; \
set -x; git push --follow-tags; set +x; \
\
echo "Publish '$$version' as a Github release? [ENTER to continue]"; \
read pass; \
. ./.env; \
curl \
--user "$$GITHUB_USER:$$GITHUB_TOKEN" \
--request POST \
--header "Accept: application/vnd.github.v3+json" \
--data "{\"tag_name\":\"$$version\", \"target_commitish\": \"$$branch\", \"name\": \"$$version\", \"body\": \"See [CHANGELOG](CHANGELOG.md) for changes and release notes.\"}" \
https://api.github.com/repos/halaxa/json-machine/releases \
;\
docker-run: ## Run a command in a latest JSON Machine PHP docker container. Ex.: make docker-run CMD="php -v"
@$(call DOCKER_RUN,$(LATEST_PHP),$(CMD))
PK ! l)0 test/bootstrap.phpnu [ createParser($json, $jsonPointer) as $key => $value) {
$result[] = [$key => $value];
}
$this->assertSame($expectedResult, $result);
}
public function data_testSyntax()
{
return [
['', '{}', []],
['', '{"a": "b"}', [['a' => 'b']]],
['', '{"a":{"b":{"c":1}}}', [['a' => ['b' => ['c' => 1]]]]],
['', '[]', []],
['', '[null,true,false,"a",0,1,42.5]', [[0 => null], [1 => true], [2 => false], [3 => 'a'], [4 => 0], [5 => 1], [6 => 42.5]]],
['', '[{"c":1}]', [[['c' => 1]]]],
['', '[{"c":1},"string",{"d":2},false]', [[0 => ['c' => 1]], [1 => 'string'], [2 => ['d' => 2]], [3 => false]]],
['', '[false,{"c":1},"string",{"d":2}]', [[0 => false], [1 => ['c' => 1]], [2 => 'string'], [3 => ['d' => 2]]]],
['', '[{"c":1,"d":2}]', [[['c' => 1, 'd' => 2]]]],
['/', '{"":{"c":1,"d":2}}', [['c' => 1], ['d' => 2]]],
['/~0', '{"~":{"c":1,"d":2}}', [['c' => 1], ['d' => 2]]],
['/~1', '{"/":{"c":1,"d":2}}', [['c' => 1], ['d' => 2]]],
['/~01', '{"~1":{"c":1,"d":2}}', [['c' => 1], ['d' => 2]]],
['/~00', '{"~0":{"c":1,"d":2}}', [['c' => 1], ['d' => 2]]],
['/path', '{"path":{"c":1,"d":2}}', [['c' => 1], ['d' => 2]]],
['/path', '{"no":[null], "path":{"c":1,"d":2}}', [['c' => 1], ['d' => 2]]],
['/0', '[{"c":1,"d":2}, [null]]', [['c' => 1], ['d' => 2]]],
['/0/path', '[{"path":{"c":1,"d":2}}]', [['c' => 1], ['d' => 2]]],
['/1/path', '[[null], {"path":{"c":1,"d":2}}]', [['c' => 1], ['d' => 2]]],
['/path/0', '{"path":[{"c":1,"d":2}, [null]]}', [['c' => 1], ['d' => 2]]],
['/path/1', '{"path":[null,{"c":1,"d":2}, [null]]}', [['c' => 1], ['d' => 2]]],
['/path/to', '{"path":{"to":{"c":1,"d":2}}}', [['c' => 1], ['d' => 2]]],
['/path/after-vector', '{"path":{"array":[],"after-vector":{"c":1,"d":2}}}', [['c' => 1], ['d' => 2]]],
['/path/after-vector', '{"path":{"array":["item"],"after-vector":{"c":1,"d":2}}}', [['c' => 1], ['d' => 2]]],
['/path/after-vector', '{"path":{"object":{"item":null},"after-vector":{"c":1,"d":2}}}', [['c' => 1], ['d' => 2]]],
['/path/after-vectors', '{"path":{"array":[],"object":{},"after-vectors":{"c":1,"d":2}}}', [['c' => 1], ['d' => 2]]],
['/0/0', '[{"0":{"c":1,"d":2}}]', [['c' => 1], ['d' => 2]]],
['/1/1', '[0,{"1":{"c":1,"d":2}}]', [['c' => 1], ['d' => 2]]],
'PR-19-FIX' => ['/datafeed/programs/1', file_get_contents(__DIR__.'/PR-19-FIX.json'), [['program_info' => ['id' => 'X1']]]],
'ISSUE-41-FIX' => ['/path', '{"path":[{"empty":{}},{"value":1}]}', [[['empty' => []]], [1 => ['value' => 1]]]],
['/-', '[{"one": 1,"two": 2},{"three": 3,"four": 4}]', [['one' => 1], ['two' => 2], ['three' => 3], ['four' => 4]]],
['/zero/-', '{"zero":[{"one": 1,"two": 2},{"three": 3,"four": 4}]}', [['one' => 1], ['two' => 2], ['three' => 3], ['four' => 4]]],
['/zero/-/three', '{"zero":[{"one": 1,"two": 2},{"three": 3,"four": 4}]}', [['three' => 3]]],
'ISSUE-62#1' => ['/-/id', '[ {"id":125}, {"id":785}, {"id":459}, {"id":853} ]', [['id' => 125], ['id' => 785], ['id' => 459], ['id' => 853]]],
'ISSUE-62#2' => ['/key/-/id', '{"key": [ {"id":125}, {"id":785}, {"id":459}, {"id":853} ]}', [['id' => 125], ['id' => 785], ['id' => 459], ['id' => 853]]],
[
['/meta_data', '/data/companies'],
'{"meta_data": {"total_rows": 2},"data": {"type": "companies","companies": [{"id": "1","company": "Company 1"},{"id": "2","company": "Company 2"}]}}',
[
['total_rows' => 2],
['0' => ['id' => '1', 'company' => 'Company 1']],
['1' => ['id' => '2', 'company' => 'Company 2']],
],
],
[
['/-/id', '/-/company'],
'[{"id": "1","company": "Company 1"},{"id": "2","company": "Company 2"}]',
[
['id' => '1'],
['company' => 'Company 1'],
['id' => '2'],
['company' => 'Company 2'],
],
],
[
['/-/id', '/0/company'],
'[{"id": "1","company": "Company 1"},{"id": "2","company": "Company 2"}]',
[
['id' => '1'],
['company' => 'Company 1'],
['id' => '2'],
],
],
];
}
/**
* @dataProvider data_testThrowsOnNotFoundJsonPointer
*
* @param string $json
* @param string $jsonPointer
*/
public function testThrowsOnNotFoundJsonPointer($json, $jsonPointer)
{
$parser = $this->createParser($json, $jsonPointer);
$this->expectException(PathNotFoundException::class);
$this->expectExceptionMessage("Paths '".implode(', ', (array) $jsonPointer)."' were not found in json stream.");
iterator_to_array($parser);
}
public function data_testThrowsOnNotFoundJsonPointer()
{
return [
'non existing pointer' => ['{}', '/not/found'],
"empty string should not match '0'" => ['{"0":[]}', '/'],
'empty string should not match 0' => ['[[]]', '/'],
'0 should not match empty string' => ['{"":[]}', '/0'],
];
}
/**
* @dataProvider data_testSyntaxError
*
* @param string $malformedJson
*/
public function testSyntaxError($malformedJson)
{
$this->expectException(SyntaxErrorException::class);
iterator_to_array($this->createParser($malformedJson));
}
public function data_testSyntaxError()
{
return [
['[}'],
['{]'],
['null'],
['true'],
['false'],
['0'],
['100'],
['"string"'],
['}'],
[']'],
[','],
[':'],
[''],
['[null null]'],
['["string" "string"]'],
['[,"string","string"]'],
['["string",,"string"]'],
['["string","string",]'],
['["string",1eeee1]'],
['{"key\u000Z": "non hex key"}'],
];
}
/**
* @dataProvider data_testUnexpectedEndError
*
* @param string $malformedJson
*/
public function testUnexpectedEndError($malformedJson)
{
$this->expectException(UnexpectedEndSyntaxErrorException::class);
iterator_to_array($this->createParser($malformedJson));
}
public function data_testUnexpectedEndError()
{
return [
['['],
['{'],
['["string"'],
['["string",'],
['[{"string":"string"}'],
['[{"string":"string"},'],
['[{"string":"string"},{'],
['[{"string":"string"},{"str'],
['[{"string":"string"},{"string"'],
['{"string"'],
['{"string":'],
['{"string":"string"'],
['{"string":["string","string"]'],
['{"string":["string","string"'],
['{"string":["string","string",'],
['{"string":["string","string","str'],
];
}
public function testGeneratorQuitsAfterFirstFoundCollectionHasBeenFinished()
{
$json = '
{
"results": [1],
"other": [2],
"results": [3]
}
';
$parser = $this->createParser($json, '/results');
$this->assertSame([1], iterator_to_array($parser));
}
public function testScalarResult()
{
$result = $this->createParser('{"result":{"items": [1,2,3],"count": 3}}', '/result/count');
$this->assertSame([3], iterator_to_array($result));
}
public function testScalarResultInArray()
{
$result = $this->createParser('{"result":[1,2,3]}', '/result/0');
$this->assertSame([1], iterator_to_array($result));
}
public function testGeneratorQuitsAfterFirstScalarHasBeenFound()
{
$json = '
{
"result": "one",
"other": [2],
"result": "three"
}
';
$parser = $this->createParser($json, '/result');
$this->assertSame(['result' => 'one'], iterator_to_array($parser));
}
public function testGeneratorYieldsNestedValues()
{
$json = '
{
"zero": [
{
"one": "ignored",
"two": [
{
"three": 1
}
],
"four": [
{
"five": "ignored"
}
]
},
{
"one": 1,
"two": [
{
"three": 2
},
{
"three": 3
}
],
"four": [
{
"five": "ignored"
}
]
}
]
}
';
$parser = $this->createParser($json, '/zero/-/two/-/three');
$i = 0;
$expectedKey = 'three';
$expectedValues = [1, 2, 3];
foreach ($parser as $key => $value) {
$this->assertSame($expectedKey, $key);
$this->assertSame($expectedValues[$i++], $value);
}
}
private function createParser($json, $jsonPointer = '')
{
return new Parser(new Tokens(new \ArrayIterator([$json])), $jsonPointer, new ExtJsonDecoder(true));
}
public function testDefaultDecodingStructureIsObject()
{
$items = new Parser(new Tokens(new StringChunks('[{"key": "value"}]')));
foreach ($items as $item) {
$this->assertEquals((object) ['key' => 'value'], $item);
}
}
/**
* @dataProvider data_testGetCurrentJsonPointer
*/
public function testGetCurrentJsonPointer($jsonPointer, string $json, array $currentJsonPointers)
{
$parser = $this->createParser($json, $jsonPointer);
$i = 0;
foreach ($parser as $value) {
$this->assertEquals($currentJsonPointers[$i++], $parser->getCurrentJsonPointer());
}
}
public function data_testGetCurrentJsonPointer()
{
return [
['', '{"c":1,"d":2}', ['', '']],
['/', '{"":{"c":1,"d":2}}', ['/', '/']],
['/~0', '{"~":{"c":1,"d":2}}', ['/~0', '/~0']],
['/~1', '{"/":{"c":1,"d":2}}', ['/~1', '/~1']],
['/~01', '{"~1":{"c":1,"d":2}}', ['/~01', '/~01']],
['/~00', '{"~0":{"c":1,"d":2}}', ['/~00', '/~00']],
['/~1/c', '{"/":{"c":[1,2],"d":2}}', ['/~1/c', '/~1/c']],
['/0', '[{"c":1,"d":2}, [null]]', ['/0', '/0']],
['/-', '[{"one": 1,"two": 2},{"three": 3,"four": 4}]', ['/0', '/0', '/1', '/1']],
[
['/two', '/four'],
'{"one": [1,11], "two": [2,22], "three": [3,33], "four": [4,44]}',
['/two', '/two', '/four', '/four'],
],
[
['/-/two', '/-/one'],
'[{"one": 1, "two": 2}, {"one": 1, "two": 2}]',
['/0/one', '/0/two', '/1/one', '/1/two'],
],
];
}
/**
* @dataProvider data_testGetMatchedJsonPointer
*/
public function testGetMatchedJsonPointer($jsonPointer, string $json, array $matchedJsonPointers)
{
$parser = $this->createParser($json, $jsonPointer);
$i = 0;
foreach ($parser as $value) {
$this->assertEquals($matchedJsonPointers[$i++], $parser->getMatchedJsonPointer());
}
}
public function data_testGetMatchedJsonPointer()
{
return [
['', '{"c":1,"d":2}', ['', '']],
['/', '{"":{"c":1,"d":2}}', ['/', '/']],
['/~0', '{"~":{"c":1,"d":2}}', ['/~0', '/~0']],
['/~1', '{"/":{"c":1,"d":2}}', ['/~1', '/~1']],
['/~01', '{"~1":{"c":1,"d":2}}', ['/~01', '/~01']],
['/~00', '{"~0":{"c":1,"d":2}}', ['/~00', '/~00']],
['/~1/c', '{"/":{"c":[1,2],"d":2}}', ['/~1/c', '/~1/c']],
['/0', '[{"c":1,"d":2}, [null]]', ['/0', '/0']],
['/-', '[{"one": 1,"two": 2},{"three": 3,"four": 4}]', ['/-', '/-', '/-', '/-']],
[
['/two', '/four'],
'{"one": [1,11], "two": [2,22], "three": [3,33], "four": [4,44]}',
['/two', '/two', '/four', '/four'],
],
[
['/-/two', '/-/one'],
'[{"one": 1, "two": 2}, {"one": 1, "two": 2}]',
['/-/one', '/-/two', '/-/one', '/-/two'],
],
];
}
public function testGetCurrentJsonPointerThrowsWhenCalledOutsideOfALoop()
{
$this->expectException(JsonMachineException::class);
$this->expectExceptionMessage('must be called inside a loop');
$parser = $this->createParser('[]');
$parser->getCurrentJsonPointer();
}
public function testGetCurrentJsonPointerReturnsLiteralJsonPointer()
{
$parser = $this->createParser('{"\"key\\\\":"value"}', ['/\"key\\\\']);
foreach ($parser as $key => $item) {
$this->assertSame('/\"key\\\\', $parser->getCurrentJsonPointer());
}
}
public function testGetMatchedJsonPointerThrowsWhenCalledOutsideOfALoop()
{
$this->expectException(JsonMachineException::class);
$this->expectExceptionMessage('must be called inside a loop');
$parser = $this->createParser('[]');
$parser->getMatchedJsonPointer();
}
public function testGetMatchedJsonPointerReturnsLiteralMatch()
{
$parser = $this->createParser('{"\"key\\\\":"value"}', ['/\"key\\\\']);
foreach ($parser as $key => $item) {
$this->assertSame('/\"key\\\\', $parser->getMatchedJsonPointer());
}
}
public function testGetJsonPointers()
{
$parser = $this->createParser('{}', ['/one', '/two']);
$this->assertSame(['/one', '/two'], $parser->getJsonPointers());
$parser = $this->createParser('{}');
$this->assertSame([''], $parser->getJsonPointers());
}
public function testJsonPointerReferenceTokenMatchesJsonMemberNameLiterally()
{
$parser = $this->createParser('{"\\"key":"value"}', ['/\\"key']);
foreach ($parser as $key => $item) {
$this->assertSame('"key', $key);
$this->assertSame('value', $item);
}
}
public function testGetPositionReturnsCorrectPositionWithDebugEnabled()
{
$parser = new Parser(new TokensWithDebugging(['[ 1, "two", false ]']));
$expectedPosition = [5, 12, 19];
$this->assertSame(0, $parser->getPosition());
foreach ($parser as $index => $item) {
$this->assertSame($expectedPosition[$index], $parser->getPosition(), "index:$index, item:$item");
}
$this->assertSame(21, $parser->getPosition());
}
public function testGetPositionReturns0WithDebugDisabled()
{
$parser = new Parser(new Tokens(['[ 1, "two", false ]']));
$this->assertSame(0, $parser->getPosition());
foreach ($parser as $index => $item) {
$this->assertSame(0, $parser->getPosition());
}
$this->assertSame(0, $parser->getPosition());
}
public function testGetPositionThrowsIfTokensDoNotSupportGetPosition()
{
$parser = new Parser(new \ArrayObject());
$this->expectException(JsonMachineException::class);
$parser->getPosition();
}
}
PK ! @G G ' test/JsonMachineTest/FileChunksTest.phpnu [ assertSame($expectedResult, $result);
}
public function data_testGeneratorYieldsFileChunks()
{
return [
[5, ['{"pat', 'h": {', '"key"', ':"val', 'ue"}}', "\n"]],
[6, ['{"path', '": {"k', 'ey":"v', 'alue"}', '}'."\n"]],
[1024, ['{"path": {"key":"value"}}'."\n"]],
];
}
}
PK ! Z Z ( test/JsonMachineTest/formatted-crlf.jsonnu [ {
"id": 54640519019642880,
"user": {
"notifications": null
},
"geo": "test"
}
PK ! oˇ . test/JsonMachineTest/ValidJsonPointersTest.phpnu [ expectException(InvalidArgumentException::class);
$this->expectExceptionMessage("'$jsonPointers[0]', '$jsonPointers[1]'");
(new ValidJsonPointers($jsonPointers))->toArray();
}
public function data_testThrowsOnIntersectingPaths()
{
return [
[['/companies/-/id', '/companies/0/id']],
[['/companies/-/id', '', '/companies/0/id']],
[['/companies/-/id', '']],
[['/companies/0/id', '']],
[['//in-empty-string-key', '/']],
[['/~0~1/in-escaped-key', '/~0~1']],
];
}
/**
* @dataProvider data_testThrowsOnMalformedJsonPointer
*/
public function testThrowsOnMalformedJsonPointer(array $jsonPointer)
{
$this->expectException(InvalidArgumentException::class);
$this->expectExceptionMessage('not valid');
(new ValidJsonPointers($jsonPointer))->toArray();
}
public function data_testThrowsOnMalformedJsonPointer()
{
return [
[['apple']],
[['/apple/~']],
[['apple/pie']],
[['apple/pie/']],
[[' /apple/pie/']],
[[
'/valid',
'/valid/-',
'inv/alid',
]],
];
}
public function testThrowsOnDuplicatePaths()
{
$this->expectException(InvalidArgumentException::class);
$this->expectExceptionMessage("'/one', '/one'");
(new ValidJsonPointers(['/one', '/one']))->toArray();
}
public function testToArrayReturnsJsonPointers()
{
$this->assertSame(
['/one', '/two'],
(new ValidJsonPointers(['/one', '/two']))->toArray()
);
}
}
PK ! k # test/JsonMachineTest/PR-19-FIX.jsonnu [ {
"datafeed": {
"info": {
"category": "Category name"
},
"programs": [
{
"program_info": {
"id": "X0"
}
},
{
"program_info": {
"id": "X1"
}
}
]
}
}
PK ! K] ] ) test/JsonMachineTest/StreamChunksTest.phpnu [ expectException(InvalidArgumentException::class);
new StreamChunks(false);
}
public function testGeneratorYieldsData()
{
$result = iterator_to_array(new StreamChunks(fopen('data://text/plain,test', 'r')));
$this->assertSame(['test'], $result);
}
}
PK ! ϔ` ) test/JsonMachineTest/ItemsOptionsTest.phpnu [ toArray();
$this->assertEquals($this->defaultOptions(), $optionsArray);
}
public function testHasArrayAccess()
{
$options = new ItemsOptions();
$this->assertTrue(isset($options['debug']));
$this->assertFalse($options['debug']);
}
/**
* @dataProvider defaultOptionNames
*/
public function testThrowsOnInvalidOptionType($optionName)
{
$this->expectException(InvalidArgumentException::class);
new ItemsOptions([$optionName => new InvalidValue()]);
}
public function defaultOptionNames()
{
foreach ($this->defaultOptions() as $name => $ignore) {
yield [$name];
}
}
private function defaultOptions()
{
return [
'pointer' => '',
'decoder' => new ExtJsonDecoder(),
'debug' => false,
];
}
public function testThrowsOnUnknownOption()
{
$this->expectException(InvalidArgumentException::class);
new ItemsOptions(['invalid_option_name' => 'value']);
}
}
class InvalidValue
{
}
PK ! = ( test/JsonMachineTest/AutoloadingTest.phpnu [ createAutoloadableClass();
register_shutdown_function(function () use ($dummyFile) {
@unlink($dummyFile);
});
$autoloadersBackup = $this->unregisterCurrentAutoloaders();
$autoloader = require __DIR__.'/../../src/autoloader.php';
spl_autoload_register($autoloader);
$autoloaded = class_exists('JsonMachine\\AutoloadStub');
spl_autoload_unregister($autoloader);
$this->registerPreviousAutoloaders($autoloadersBackup);
$this->assertTrue($autoloaded);
}
/**
* @runInSeparateProcess
*/
public function testIgnoresInvalidBaseNamespace()
{
$dummyFile = $this->createAutoloadableClass();
register_shutdown_function(function () use ($dummyFile) {
@unlink($dummyFile);
});
$autoloadersBackup = $this->unregisterCurrentAutoloaders();
$autoloader = require __DIR__.'/../../src/autoloader.php';
spl_autoload_register($autoloader);
$autoloaded = class_exists('XXXsonMachine\\AutoloadStub');
spl_autoload_unregister($autoloader);
$this->registerPreviousAutoloaders($autoloadersBackup);
$this->assertFalse($autoloaded);
}
private function createAutoloadableClass(): string
{
$dummyFile = __DIR__.'/../../src/AutoloadStub.php';
file_put_contents($dummyFile, ' $args[1],
'decoder' => $args[2],
'debug' => $args[3],
],
]);
$this->assertSame($expected, iterator_to_array($iterator));
}
public function testItemsYieldsObjectItemsByDefault()
{
$iterator = Items::fromString('{"path": {"key":"value"}}');
foreach ($iterator as $item) {
$this->assertEquals((object) ['key' => 'value'], $item);
}
}
public function data_testFactories()
{
$extJsonResult = ['key' => 'value'];
$passThruResult = ['"key"' => '"value"'];
$ptDecoder = new PassThruDecoder();
foreach ([true, false] as $debug) {
foreach ([
[$extJsonResult, 'fromStream', fopen('data://text/plain,{"path": {"key":"value"}}', 'r'), '/path', null, $debug],
[$extJsonResult, 'fromString', '{"path": {"key":"value"}}', '/path', null, $debug],
[$extJsonResult, 'fromFile', __DIR__.'/ItemsTest.json', '/path', null, $debug],
[$extJsonResult, 'fromIterable', ['{"path": {"key', '":"value"}}'], '/path', null, $debug],
[$extJsonResult, 'fromIterable', new \ArrayIterator(['{"path": {"key', '":"value"}}']), '/path', null, $debug],
[$passThruResult, 'fromStream', fopen('data://text/plain,{"path": {"key":"value"}}', 'r'), '/path', $ptDecoder, $debug],
[$passThruResult, 'fromString', '{"path": {"key":"value"}}', '/path', $ptDecoder, $debug],
[$passThruResult, 'fromFile', __DIR__.'/ItemsTest.json', '/path', $ptDecoder, $debug],
[$passThruResult, 'fromIterable', ['{"path": {"key', '":"value"}}'], '/path', $ptDecoder, $debug],
[$passThruResult, 'fromIterable', new \ArrayIterator(['{"path": {"key', '":"value"}}']), '/path', $ptDecoder, $debug],
] as $case) {
yield $case;
}
}
}
public function testGetPositionDebugEnabled()
{
$expectedPosition = ['key1' => 9, 'key2' => 19];
$items = Items::fromString('{"key1":1, "key2":2} ', ['debug' => true]);
foreach ($items as $key => $val) {
$this->assertSame($expectedPosition[$key], $items->getPosition());
}
}
public function testIterationWithoutForeach()
{
$iterator =
Items::fromString('{"key1":1, "key2":2}')
->getIterator();
$iterator->rewind();
$this->assertTrue($iterator->valid());
$this->assertSame(['key1', 1], [$iterator->key(), $iterator->current()]);
$iterator->next();
$this->assertTrue($iterator->valid());
$this->assertSame(['key2', 2], [$iterator->key(), $iterator->current()]);
$iterator->next();
$this->assertFalse($iterator->valid());
}
public function testIsDebugEnabled()
{
$items = $iterator = Items::fromString('{}');
$this->assertFalse($items->isDebugEnabled());
$items = $iterator = Items::fromString('{}', ['debug' => true]);
$this->assertTrue($items->isDebugEnabled());
}
public function testGetCurrentJsonPointer()
{
$items = $iterator = Items::fromString(
'[{"two": 2, "one": 1}]',
['pointer' => ['/-/one', '/-/two']]
);
$iterator = $items->getIterator();
$iterator->rewind();
$iterator->current();
$this->assertSame('/0/two', $items->getCurrentJsonPointer());
$iterator->next();
$iterator->current();
$this->assertSame('/0/one', $items->getCurrentJsonPointer());
}
public function testGetMatchedJsonPointer()
{
$items = $iterator = Items::fromString(
'[{"two": 2, "one": 1}]',
['pointer' => ['/-/one', '/-/two']]
);
$iterator = $items->getIterator();
$iterator->rewind();
$iterator->current();
$this->assertSame('/-/two', $items->getMatchedJsonPointer());
$iterator->next();
$iterator->current();
$this->assertSame('/-/one', $items->getMatchedJsonPointer());
}
public function testGetJsonPointers()
{
$items = Items::fromString('[]', ['pointer' => ['/one', '/two']]);
$this->assertSame(['/one', '/two'], $items->getJsonPointers());
}
}
PK ! , ; test/JsonMachineTest/Exception/SyntaxErrorExceptionTest.phpnu [ assertContains('msg 42', $exception->getMessage());
$this->assertContains('24', $exception->getMessage());
}
}
PK ! xZ Z & test/JsonMachineTest/formatted-cr.jsonnu [ {
"id": 54640519019642880,
"user": {
"notifications": null
},
"geo": "test"
}
PK ! Z Z & test/JsonMachineTest/formatted-lf.jsonnu [ {
"id": 54640519019642880,
"user": {
"notifications": null
},
"geo": "test"
}
PK ! s5G G 7 test/JsonMachineTest/JsonDecoder/ExtJsonDecoderTest.phpnu [ decode($json);
$this->assertTrue('object' === gettype($defaultResult->getValue()));
$this->assertFalse('string' === gettype($defaultResult->getValue()->bigint));
$this->assertSame([['deeper']], $defaultResult->getValue()->deep);
}
public function testPassesAssocTrueOptionToJsonDecode()
{
$json = '{"bigint": 123456789123456789123456789, "deep": [["deeper"]]}';
$assocDecoder = new ExtJsonDecoder(true);
$assocResult = $assocDecoder->decode($json);
$this->assertTrue('array' === gettype($assocResult->getValue()));
}
public function testPassesAssocFalseOptionToJsonDecode()
{
$json = '{"bigint": 123456789123456789123456789, "deep": [["deeper"]]}';
$objDecoder = new ExtJsonDecoder(false);
$objResult = $objDecoder->decode($json);
$this->assertTrue('object' === gettype($objResult->getValue()));
}
public function testPassesPassesDepthOptionToJsonDecode()
{
$json = '{"bigint": 123456789123456789123456789, "deep": [["deeper"]]}';
$depthDecoder = new ExtJsonDecoder(true, 1);
$depthResult = $depthDecoder->decode($json);
$this->assertFalse($depthResult->isOk());
$this->assertSame('Maximum stack depth exceeded', $depthResult->getErrorMessage());
}
public function testPassesPassesBigIntOptionToJsonDecode()
{
$bigintDecoder = new ExtJsonDecoder(false, 1, JSON_BIGINT_AS_STRING);
$bigintResult = $bigintDecoder->decode('123123123123123123123');
$this->assertSame('123123123123123123123', $bigintResult->getValue());
}
}
PK ! ٕ 6 test/JsonMachineTest/JsonDecoder/DecodingErrorTest.phpnu [ assertSame('"json\"', $decodingError->getMalformedJson());
}
public function testGetErrorMessage()
{
$decodingError = new DecodingError('', 'something bad happened');
$this->assertSame('something bad happened', $decodingError->getErrorMessage());
}
}
PK ! Ϋ 4 test/JsonMachineTest/JsonDecoder/ValidResultTest.phpnu [ assertSame('Value X', $result->getValue());
}
public function testIsOk()
{
$result = new ValidResult('X');
$this->assertTrue($result->isOk());
}
}
PK ! N) ) 6 test/JsonMachineTest/JsonDecoder/InvalidResultTest.phpnu [ assertSame('Error X', $result->getErrorMessage());
}
public function testIsOk()
{
$result = new InvalidResult('X');
$this->assertFalse($result->isOk());
}
}
PK ! _! = test/JsonMachineTest/JsonDecoder/ErrorWrappingDecoderTest.phpnu [ decode('"json"');
$this->assertTrue($result->isOk());
$this->assertEquals($case['wrappedResult'], $result);
}
public function data_testCorrectlyWrapsResults()
{
$notOkResult = new InvalidResult('Error happened.');
$okResult = new ValidResult('json');
$wrappedNotOkResult = new ValidResult(new DecodingError('"json"', 'Error happened.'));
$wrappedOkResult = $okResult;
return [
[
[
'result' => $notOkResult,
'wrappedResult' => $wrappedNotOkResult,
],
],
[
[
'result' => $okResult,
'wrappedResult' => $wrappedOkResult,
],
],
];
}
public function testCatchesErrorInsideIteratedJsonChunk()
{
$json = /* @lang JSON */ '
{
"results": [
{"correct": "correct"},
{"incorrect": nulll},
{"correct": "correct"}
]
}
';
$items = Items::fromString($json, [
'pointer' => '/results',
'decoder' => new ErrorWrappingDecoder(new ExtJsonDecoder(true)),
]);
$result = iterator_to_array($items);
$this->assertSame('correct', $result[0]['correct']);
$this->assertSame('correct', $result[2]['correct']);
/** @var DecodingError $decodingError */
$decodingError = $result[1];
$this->assertInstanceOf(DecodingError::class, $decodingError);
$this->assertSame('{"incorrect":nulll}', $decodingError->getMalformedJson());
$this->assertSame('Syntax error', $decodingError->getErrorMessage());
}
}
PK ! B 8 test/JsonMachineTest/JsonDecoder/PassThruDecoderTest.phpnu [ decode('["json"]');
$this->assertSame('["json"]', $passThruResult->getValue());
}
}
PK ! ʢ?c c 0 test/JsonMachineTest/JsonDecoder/StubDecoder.phpnu [ decoded = $decoded;
}
public function decode($jsonValue)
{
return $this->decoded;
}
}
PK ! m?I I ) test/JsonMachineTest/StringChunksTest.phpnu [ assertSame($expectedResult, $result);
}
public function data_testGeneratorYieldsStringChunks()
{
return [
// single-byte:
['onetwo', 6, ['onetwo']],
['onetwo', 7, ['onetwo']],
['onetwo', 3, ['one', 'two']],
['onetwo', 4, ['onet', 'wo']],
];
}
}
PK ! :F7 7 # test/JsonMachineTest/TokensTest.phpnu [ [TokensWithDebugging::class],
'debug disabled' => [Tokens::class],
];
}
/**
* @dataProvider bothDebugModes
*/
public function testCorrectlyYieldsZeroToken($tokensClass)
{
$data = ['0'];
$expected = ['0'];
$this->assertEquals($expected, iterator_to_array(new $tokensClass(new \ArrayIterator($data))));
$stream = fopen('data://text/plain,{"value":0}', 'r');
$expected = ['{', '"value"', ':', '0', '}'];
$this->assertEquals($expected, iterator_to_array(new $tokensClass(new StreamChunks($stream, 10))));
}
/**
* @dataProvider bothDebugModes
*/
public function testGeneratesTokens($tokensClass)
{
$data = ['{}[],:null,"string" false:', 'true,1,100000,1.555{-56]"","\\""'];
$expected = ['{', '}', '[', ']', ',', ':', 'null', ',', '"string"', 'false', ':', 'true', ',', '1', ',', '100000', ',', '1.555', '{', '-56', ']', '""', ',', '"\\""'];
$this->assertEquals($expected, iterator_to_array(new $tokensClass(new \ArrayIterator($data))));
}
/**
* @dataProvider bothDebugModes
*/
public function testWithBOM($tokensClass)
{
$data = ["\xEF\xBB\xBF".'{}'];
$expected = ['{', '}'];
$this->assertEquals($expected, iterator_to_array(new $tokensClass(new \ArrayIterator($data))));
}
/**
* @dataProvider bothDebugModes
*/
public function testCorrectlyParsesTwoBackslashesAtTheEndOfAString($tokensClass)
{
$this->assertEquals(['"test\\\\"', ':'], iterator_to_array(new $tokensClass(new \ArrayIterator(['"test\\\\":']))));
}
/**
* @dataProvider bothDebugModes
*/
public function testCorrectlyParsesEscapedQuotesInTheMiddleOfAString($tokensClass)
{
$json = '"test\"test":';
$expected = ['"test\"test"', ':'];
$this->assertEquals($expected, iterator_to_array(new $tokensClass(new \ArrayIterator([$json]))));
}
/**
* @dataProvider bothDebugModes
*/
public function testCorrectlyParsesChunksSplitBeforeStringEnd($tokensClass)
{
$chunks = ['{"path": {"key":"value', '"}}'];
$expected = ['{', '"path"', ':', '{', '"key"', ':', '"value"', '}', '}'];
$this->assertEquals($expected, iterator_to_array(new $tokensClass(new \ArrayIterator($chunks))));
}
/**
* @dataProvider bothDebugModes
*/
public function testCorrectlyParsesChunksSplitBeforeEscapedCharacter($tokensClass)
{
$chunks = ['{"path": {"key":"value\\', '""}}'];
$expected = ['{', '"path"', ':', '{', '"key"', ':', '"value\""', '}', '}'];
$this->assertEquals($expected, iterator_to_array(new $tokensClass(new \ArrayIterator($chunks))));
}
/**
* @dataProvider bothDebugModes
*/
public function testCorrectlyParsesChunksSplitAfterEscapedCharacter($tokensClass)
{
$chunks = ['{"path": {"key":"value\\"', '"}}'];
$expected = ['{', '"path"', ':', '{', '"key"', ':', '"value\""', '}', '}'];
$this->assertEquals($expected, iterator_to_array(new $tokensClass(new \ArrayIterator($chunks))));
}
/**
* @dataProvider bothDebugModes
*/
public function testAnyPossibleChunkSplit($tokensClass)
{
$json = '
{
"datafeed": {
"info": {
"category": "Category name"
},
"programs": [
{
"program_info": {
"id": "X0\"\\\\",
"number": 123,
"constant": false
}
},
{
"program_info": {
"id": "\b\f\n\r\t\u0020X1"
}
}
]
}
}
';
$expected = [
'{', '"datafeed"', ':', '{', '"info"', ':', '{', '"category"', ':', '"Category name"', '}', ',',
'"programs"', ':', '[', '{', '"program_info"', ':', '{', '"id"', ':', '"X0\\"\\\\"', ',', '"number"', ':',
'123', ',', '"constant"', ':', 'false', '}', '}', ',', '{', '"program_info"', ':', '{', '"id"', ':',
'"\b\f\n\r\t\u0020X1"', '}', '}', ']', '}', '}',
];
foreach (range(1, strlen($json)) as $chunkLength) {
$chunks = str_split($json, $chunkLength);
$result = iterator_to_array(new $tokensClass($chunks));
$this->assertSame($expected, $result);
}
}
/**
* @dataProvider jsonFilesWithDifferentLineEndings
*/
public function testProvidesLocationalDataWhenDebugEnabled(string $jsonFilePath)
{
$jsonFileContents = file_get_contents($jsonFilePath);
$tokens = new TokensWithDebugging(new StringChunks($jsonFileContents));
$expectedTokens = $this->expectedTokens();
$i = 0;
foreach ($tokens as $token) {
++$i;
$expectedToken = array_shift($expectedTokens);
$this->assertEquals($expectedToken[0], $token, 'token failed with expected token #'.$i);
$this->assertEquals($expectedToken[1], $tokens->getLine(), 'line failed with expected token #'.$i);
$this->assertEquals($expectedToken[2], $tokens->getColumn(), 'column failed with expected token #'.$i);
}
}
/**
* @dataProvider jsonFilesWithDifferentLineEndings
*/
public function testProvidesLocationalDataWhenDebugDisabled(string $jsonFilePath)
{
$tokens = new Tokens(new FileChunks($jsonFilePath));
$expectedTokens = $this->expectedTokens();
$i = 0;
foreach ($tokens as $token) {
++$i;
$expectedToken = array_shift($expectedTokens);
$this->assertEquals($expectedToken[0], $token, 'token failed with expected token #'.$i);
$this->assertEquals(1, $tokens->getLine(), 'line failed with expected token #'.$i);
$this->assertEquals(0, $tokens->getColumn(), 'column failed with expected token #'.$i);
}
}
public function testGetPositionWthDebugging()
{
$tokens = new TokensWithDebugging(['[ 1, "two", false ]']);
$expectedPosition = [1, 5, 6, 12, 13, 19, 21];
$this->assertSame(0, $tokens->getPosition());
foreach ($tokens as $index => $item) {
$this->assertSame($expectedPosition[$index], $tokens->getPosition(), "index:$index, item:$item");
}
$this->assertSame(21, $tokens->getPosition());
}
public function testGetPositionNoDebugging()
{
$tokens = new Tokens(['[ 1, "two", false ]']);
$this->assertSame(0, $tokens->getPosition());
foreach ($tokens as $index => $item) {
$this->assertSame(0, $tokens->getPosition(), "index:$index, item:$item");
}
$this->assertSame(0, $tokens->getPosition());
}
public function jsonFilesWithDifferentLineEndings()
{
return [
'cr new lines' => [__DIR__.'/formatted-cr.json'],
'lf new lines' => [__DIR__.'/formatted-lf.json'],
'crlf new lines' => [__DIR__.'/formatted-crlf.json'],
];
}
private function expectedTokens()
{
return [
// lexeme, line, column
['{', 1, 1],
['"id"', 2, 3],
[':', 2, 7],
['54640519019642880', 2, 9],
[',', 2, 26],
['"user"', 3, 3],
[':', 3, 9],
['{', 3, 11],
['"notifications"', 4, 5],
[':', 4, 20],
['null', 4, 22],
['}', 5, 3],
[',', 5, 4],
['"geo"', 6, 3],
[':', 6, 8],
['"test"', 6, 10],
['}', 7, 1],
];
}
}
PK ! # test/JsonMachineTest/ItemsTest.jsonnu [ {"path": {"key":"value"}}
PK ! q $ test/performance/testPerformance.phpnu [ function ($file) {
return Items::fromFile($file);
},
'Items::fromString()' => function ($file) {
return Items::fromString(stream_get_contents(fopen($file, 'r')));
},
'Items::fromFile() - debug' => function ($file) {
return Items::fromFile($file, ['debug' => true]);
},
'Items::fromString() - debug' => function ($file) {
return Items::fromString(stream_get_contents(fopen($file, 'r')), ['debug' => true]);
},
'json_decode()' => function ($file) {
return json_decode(stream_get_contents(fopen($file, 'r')), true);
},
];
$tmpJsonFileName = createBigJsonFile();
$fileSizeMb = (filesize($tmpJsonFileName) / 1024 / 1024);
echo 'File size: '.round($fileSizeMb, 2),' MB'.PHP_EOL;
foreach ($decoders as $name => $decoder) {
$start = microtime(true);
$result = $decoder($tmpJsonFileName);
if ( ! $result instanceof \Traversable && ! is_array($result)) {
$textResult = 'Decoding error';
} else {
foreach ($result as $key => $item) {
}
$time = microtime(true) - $start;
$textResult = round($fileSizeMb / $time, 2).' MB/s';
}
echo str_pad($name.': ', 37, '.')." $textResult".PHP_EOL;
}
@unlink($tmpJsonFileName);
function createBigJsonFile()
{
$tmpJson = tempnam(sys_get_temp_dir(), 'json_');
$f = fopen($tmpJson, 'w');
$separator = '';
fputs($f, '[');
for ($i = 0; $i < 6000; ++$i) {
fputs($f, $separator);
fputs($f, file_get_contents(__DIR__.'/twitter_example_'.($i % 2).'.json'));
$separator = ",\n\n";
}
fputs($f, ']');
fclose($f);
return $tmpJson;
}
PK ! = ' test/performance/twitter_example_1.jsonnu [ {
"text": "RT @PostGradProblem: In preparation for the NFL lockout, I will be spending twice as much time analyzing my fantasy baseball team during ...",
"truncated": true,
"in_reply_to_user_id": null,
"in_reply_to_status_id": null,
"favorited": false,
"source": "Twitter for iPhone",
"in_reply_to_screen_name": null,
"in_reply_to_status_id_str": null,
"id_str": "54691802283900928",
"entities": {
"user_mentions": [
{
"indices": [
3,
19
],
"screen_name": "PostGradProblem",
"id_str": "271572434",
"name": "PostGradProblems",
"id": 271572434
}
],
"urls": [ ],
"hashtags": [ ]
},
"contributors": null,
"retweeted": false,
"in_reply_to_user_id_str": null,
"place": null,
"retweet_count": 4,
"created_at": "Sun Apr 03 23:48:36 +0000 2011",
"retweeted_status": {
"text": "In preparation for the NFL lockout, I will be spending twice as much time analyzing my fantasy baseball team during company time. #PGP",
"truncated": false,
"in_reply_to_user_id": null,
"in_reply_to_status_id": null,
"favorited": false,
"source": "HootSuite",
"in_reply_to_screen_name": null,
"in_reply_to_status_id_str": null,
"id_str": "54640519019642881",
"entities": {
"user_mentions": [ ],
"urls": [ ],
"hashtags": [
{
"text": "PGP",
"indices": [
130,
134
]
}
]
},
"contributors": null,
"retweeted": false,
"in_reply_to_user_id_str": null,
"place": null,
"retweet_count": 4,
"created_at": "Sun Apr 03 20:24:49 +0000 2011",
"user": {
"notifications": null,
"profile_use_background_image": true,
"statuses_count": 31,
"profile_background_color": "C0DEED",
"followers_count": 3066,
"profile_image_url": "http://a2.twimg.com/profile_images/1285770264/PGP_normal.jpg",
"listed_count": 6,
"profile_background_image_url": "http://a3.twimg.com/a/1301071706/images/themes/theme1/bg.png",
"description": "",
"screen_name": "PostGradProblem",
"default_profile": true,
"verified": false,
"time_zone": null,
"profile_text_color": "333333",
"is_translator": false,
"profile_sidebar_fill_color": "DDEEF6",
"location": "",
"id_str": "271572434",
"default_profile_image": false,
"profile_background_tile": false,
"lang": "en",
"friends_count": 21,
"protected": false,
"favourites_count": 0,
"created_at": "Thu Mar 24 19:45:44 +0000 2011",
"profile_link_color": "0084B4",
"name": "PostGradProblems",
"show_all_inline_media": false,
"follow_request_sent": null,
"geo_enabled": false,
"profile_sidebar_border_color": "C0DEED",
"url": null,
"id": 271572434,
"contributors_enabled": false,
"following": null,
"utc_offset": null
},
"id": 54640519019642880,
"coordinates": null,
"geo": null
},
"user": {
"notifications": null,
"profile_use_background_image": true,
"statuses_count": 351,
"profile_background_color": "C0DEED",
"followers_count": 48,
"profile_image_url": "http://a1.twimg.com/profile_images/455128973/gCsVUnofNqqyd6tdOGevROvko1_500_normal.jpg",
"listed_count": 0,
"profile_background_image_url": "http://a3.twimg.com/a/1300479984/images/themes/theme1/bg.png",
"description": "watcha doin in my waters?",
"screen_name": "OldGREG85",
"default_profile": true,
"verified": false,
"time_zone": "Hawaii",
"profile_text_color": "333333",
"is_translator": false,
"profile_sidebar_fill_color": "DDEEF6",
"location": "Texas",
"id_str": "80177619",
"default_profile_image": false,
"profile_background_tile": false,
"lang": "en",
"friends_count": 81,
"protected": false,
"favourites_count": 0,
"created_at": "Tue Oct 06 01:13:17 +0000 2009",
"profile_link_color": "0084B4",
"name": "GG",
"show_all_inline_media": false,
"follow_request_sent": null,
"geo_enabled": false,
"profile_sidebar_border_color": "C0DEED",
"url": null,
"id": 80177619,
"contributors_enabled": false,
"following": null,
"utc_offset": -36000
},
"id": 54691802283900930,
"coordinates": null,
"geo": null
}
PK ! ($2 $2 ' test/performance/twitter_example_0.jsonnu [ {
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000 2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": [
]
},
"in_reply_to_user_id_str": null,
"contributors": null,
"text": "Aggressive Ponytail #freebandnames",
"metadata": {
"iso_language_code": "en",
"result_type": "recent"
},
"retweet_count": 0,
"in_reply_to_status_id_str": null,
"id": 250075927172759552,
"geo": null,
"retweeted": false,
"in_reply_to_user_id": null,
"place": null,
"user": {
"profile_sidebar_fill_color": "DDEEF6",
"profile_sidebar_border_color": "C0DEED",
"profile_background_tile": false,
"name": "Sean Cummings",
"profile_image_url": "http://a0.twimg.com/profile_images/2359746665/1v6zfgqo8g0d3mk7ii5s_normal.jpeg",
"created_at": "Mon Apr 26 06:01:55 +0000 2010",
"location": "LA, CA",
"follow_request_sent": null,
"profile_link_color": "0084B4",
"is_translator": false,
"id_str": "137238150",
"entities": {
"url": {
"urls": [
{
"expanded_url": null,
"url": "",
"indices": [
0,
0
]
}
]
},
"description": {
"urls": [
]
}
},
"default_profile": true,
"contributors_enabled": false,
"favourites_count": 0,
"url": null,
"profile_image_url_https": "https://si0.twimg.com/profile_images/2359746665/1v6zfgqo8g0d3mk7ii5s_normal.jpeg",
"utc_offset": -28800,
"id": 137238150,
"profile_use_background_image": true,
"listed_count": 2,
"profile_text_color": "333333",
"lang": "en",
"followers_count": 70,
"protected": false,
"notifications": null,
"profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme1/bg.png",
"profile_background_color": "C0DEED",
"verified": false,
"geo_enabled": true,
"time_zone": "Pacific Time (US & Canada)",
"description": "Born 330 Live 310",
"default_profile_image": false,
"profile_background_image_url": "http://a0.twimg.com/images/themes/theme1/bg.png",
"statuses_count": 579,
"friends_count": 110,
"following": null,
"show_all_inline_media": false,
"screen_name": "sean_cummings"
},
"in_reply_to_screen_name": null,
"source": "Twitter for Mac",
"in_reply_to_status_id": null
},
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Fri Sep 21 23:40:54 +0000 2012",
"id_str": "249292149810667520",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "FreeBandNames",
"indices": [
20,
34
]
}
],
"user_mentions": [
]
},
"in_reply_to_user_id_str": null,
"contributors": null,
"text": "Thee Namaste Nerdz. #FreeBandNames",
"metadata": {
"iso_language_code": "pl",
"result_type": "recent"
},
"retweet_count": 0,
"in_reply_to_status_id_str": null,
"id": 249292149810667520,
"geo": null,
"retweeted": false,
"in_reply_to_user_id": null,
"place": null,
"user": {
"profile_sidebar_fill_color": "DDFFCC",
"profile_sidebar_border_color": "BDDCAD",
"profile_background_tile": true,
"name": "Chaz Martenstein",
"profile_image_url": "http://a0.twimg.com/profile_images/447958234/Lichtenstein_normal.jpg",
"created_at": "Tue Apr 07 19:05:07 +0000 2009",
"location": "Durham, NC",
"follow_request_sent": null,
"profile_link_color": "0084B4",
"is_translator": false,
"id_str": "29516238",
"entities": {
"url": {
"urls": [
{
"expanded_url": null,
"url": "http://bullcityrecords.com/wnng/",
"indices": [
0,
32
]
}
]
},
"description": {
"urls": [
]
}
},
"default_profile": false,
"contributors_enabled": false,
"favourites_count": 8,
"url": "http://bullcityrecords.com/wnng/",
"profile_image_url_https": "https://si0.twimg.com/profile_images/447958234/Lichtenstein_normal.jpg",
"utc_offset": -18000,
"id": 29516238,
"profile_use_background_image": true,
"listed_count": 118,
"profile_text_color": "333333",
"lang": "en",
"followers_count": 2052,
"protected": false,
"notifications": null,
"profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/9423277/background_tile.bmp",
"profile_background_color": "9AE4E8",
"verified": false,
"geo_enabled": false,
"time_zone": "Eastern Time (US & Canada)",
"description": "You will come to Durham, North Carolina. I will sell you some records then, here in Durham, North Carolina. Fun will happen.",
"default_profile_image": false,
"profile_background_image_url": "http://a0.twimg.com/profile_background_images/9423277/background_tile.bmp",
"statuses_count": 7579,
"friends_count": 348,
"following": null,
"show_all_inline_media": true,
"screen_name": "bullcityrecords"
},
"in_reply_to_screen_name": null,
"source": "web",
"in_reply_to_status_id": null
},
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Fri Sep 21 23:30:20 +0000 2012",
"id_str": "249289491129438208",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
29,
43
]
}
],
"user_mentions": [
]
},
"in_reply_to_user_id_str": null,
"contributors": null,
"text": "Mexican Heaven, Mexican Hell #freebandnames",
"metadata": {
"iso_language_code": "en",
"result_type": "recent"
},
"retweet_count": 0,
"in_reply_to_status_id_str": null,
"id": 249289491129438208,
"geo": null,
"retweeted": false,
"in_reply_to_user_id": null,
"place": null,
"user": {
"profile_sidebar_fill_color": "99CC33",
"profile_sidebar_border_color": "829D5E",
"profile_background_tile": false,
"name": "Thomas John Wakeman",
"profile_image_url": "http://a0.twimg.com/profile_images/2219333930/Froggystyle_normal.png",
"created_at": "Tue Sep 01 21:21:35 +0000 2009",
"location": "Kingston New York",
"follow_request_sent": null,
"profile_link_color": "D02B55",
"is_translator": false,
"id_str": "70789458",
"entities": {
"url": {
"urls": [
{
"expanded_url": null,
"url": "",
"indices": [
0,
0
]
}
]
},
"description": {
"urls": [
]
}
},
"default_profile": false,
"contributors_enabled": false,
"favourites_count": 19,
"url": null,
"profile_image_url_https": "https://si0.twimg.com/profile_images/2219333930/Froggystyle_normal.png",
"utc_offset": -18000,
"id": 70789458,
"profile_use_background_image": true,
"listed_count": 1,
"profile_text_color": "3E4415",
"lang": "en",
"followers_count": 63,
"protected": false,
"notifications": null,
"profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme5/bg.gif",
"profile_background_color": "352726",
"verified": false,
"geo_enabled": false,
"time_zone": "Eastern Time (US & Canada)",
"description": "Science Fiction Writer, sort of. Likes Superheroes, Mole People, Alt. Timelines.",
"default_profile_image": false,
"profile_background_image_url": "http://a0.twimg.com/images/themes/theme5/bg.gif",
"statuses_count": 1048,
"friends_count": 63,
"following": null,
"show_all_inline_media": false,
"screen_name": "MonkiesFist"
},
"in_reply_to_screen_name": null,
"source": "web",
"in_reply_to_status_id": null
},
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Fri Sep 21 22:51:18 +0000 2012",
"id_str": "249279667666817024",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": [
]
},
"in_reply_to_user_id_str": null,
"contributors": null,
"text": "The Foolish Mortals #freebandnames",
"metadata": {
"iso_language_code": "en",
"result_type": "recent"
},
"retweet_count": 0,
"in_reply_to_status_id_str": null,
"id": 249279667666817024,
"geo": null,
"retweeted": false,
"in_reply_to_user_id": null,
"place": null,
"user": {
"profile_sidebar_fill_color": "BFAC83",
"profile_sidebar_border_color": "615A44",
"profile_background_tile": true,
"name": "Marty Elmer",
"profile_image_url": "http://a0.twimg.com/profile_images/1629790393/shrinker_2000_trans_normal.png",
"created_at": "Mon May 04 00:05:00 +0000 2009",
"location": "Wisconsin, USA",
"follow_request_sent": null,
"profile_link_color": "3B2A26",
"is_translator": false,
"id_str": "37539828",
"entities": {
"url": {
"urls": [
{
"expanded_url": null,
"url": "http://www.omnitarian.me",
"indices": [
0,
24
]
}
]
},
"description": {
"urls": [
]
}
},
"default_profile": false,
"contributors_enabled": false,
"favourites_count": 647,
"url": "http://www.omnitarian.me",
"profile_image_url_https": "https://si0.twimg.com/profile_images/1629790393/shrinker_2000_trans_normal.png",
"utc_offset": -21600,
"id": 37539828,
"profile_use_background_image": true,
"listed_count": 52,
"profile_text_color": "000000",
"lang": "en",
"followers_count": 608,
"protected": false,
"notifications": null,
"profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/106455659/rect6056-9.png",
"profile_background_color": "EEE3C4",
"verified": false,
"geo_enabled": false,
"time_zone": "Central Time (US & Canada)",
"description": "Cartoonist, Illustrator, and T-Shirt connoisseur",
"default_profile_image": false,
"profile_background_image_url": "http://a0.twimg.com/profile_background_images/106455659/rect6056-9.png",
"statuses_count": 3575,
"friends_count": 249,
"following": null,
"show_all_inline_media": true,
"screen_name": "Omnitarian"
},
"in_reply_to_screen_name": null,
"source": "Twitter for iPhone",
"in_reply_to_status_id": null
}
],
"search_metadata": {
"max_id": 250126199840518145,
"since_id": 24012619984051000,
"refresh_url": "?since_id=250126199840518145&q=%23freebandnames&result_type=mixed&include_entities=1",
"next_results": "?max_id=249279667666817023&q=%23freebandnames&count=4&include_entities=1&result_type=mixed",
"count": 4,
"completed_in": 0.035,
"since_id_str": "24012619984051000",
"query": "%23freebandnames",
"max_id_str": "250126199840518145"
}
}
PK ! ac% % phpunit.xml.distnu [ PK ! 3Zh h
e .gitignorenu [ PK ! H. build/composer-lt-7.2.jsonnu [ PK ! 36T T build/docker-run.shnu ȯ PK ! c build/composer-update.shnu ȯ PK ! M build/build-image.shnu ȯ PK ! g
build/update-changelog.phpnu [ PK ! U-<- <-
img/logo.pngnu [ PK ! 6 : img/wallpaper.pngnu [ PK !
3" " img/logo.svgnu [ PK ! . . img/github.pngnu [ PK ! t N examples/guzzleHttp.phpnu [ PK ! , P examples/symfonyHttpClient.phpnu [ PK ! ;pUh h nS examples/memLeak.phpnu [ PK ! A 4
W composer.jsonnu [ PK ! 5 V] CHANGELOG.mdnu [ PK ! ᶏl l w src/autoloader.phpnu [ PK ! ԾR R
T| src/Items.phpnu [ PK ! Tr r src/ItemsOptions.phpnu [ PK ! Kj src/FileChunks.phpnu [ PK ! ʓ src/PositionAware.phpnu [ PK ! DPU src/StreamChunks.phpnu [ PK ! L$u u & src/Exception/JsonMachineException.phpnu [ PK ! 2^ * src/Exception/InvalidArgumentException.phpnu [ PK ! 8qv ' f src/Exception/PathNotFoundException.phpnu [ PK ! R~] & = src/Exception/SyntaxErrorException.phpnu [ PK ! r
3 src/Exception/UnexpectedEndSyntaxErrorException.phpnu [ PK ! wޏ ! src/JsonDecoder/DecodingError.phpnu [ PK ! d c src/JsonDecoder/ItemDecoder.phpnu [ PK ! , " С src/JsonDecoder/ExtJsonDecoder.phpnu [ PK ! !l ! % src/JsonDecoder/InvalidResult.phpnu [ PK ! 쩭 # $ src/JsonDecoder/PassThruDecoder.phpnu [ PK ! !`F F ( L src/JsonDecoder/ErrorWrappingDecoder.phpnu [ PK ! 8u src/JsonDecoder/ValidResult.phpnu [ PK ! * L L Ϭ src/Tokens.phpnu [ PK ! eT$9 $9 Y src/Parser.phpnu [ PK ! " src/StringChunks.phpnu [ PK ! # src/TokensWithDebugging.phpnu [ PK ! <
src/ValidJsonPointers.phpnu [ PK ! [yCR, R, v LICENSE.txtnu [ PK ! MD{Z Z > README.mdnu [ PK ! NK .php-cs-fixer.dist.phpnu [ PK ! .github/FUNDING.ymlnu [ PK ! [n"c c H .github/workflows/makefile.ymlnu [ PK ! R Makefilenu [ PK ! l)0 ũ test/bootstrap.phpnu [ PK ! \B B # test/JsonMachineTest/ParserTest.phpnu [ PK ! @G G '
test/JsonMachineTest/FileChunksTest.phpnu [ PK ! Z Z ( test/JsonMachineTest/formatted-crlf.jsonnu [ PK ! oˇ . ] test/JsonMachineTest/ValidJsonPointersTest.phpnu [ PK ! k # b test/JsonMachineTest/PR-19-FIX.jsonnu [ PK ! K] ] ) test/JsonMachineTest/StreamChunksTest.phpnu [ PK ! ϔ` ) e test/JsonMachineTest/ItemsOptionsTest.phpnu [ PK ! = ( test/JsonMachineTest/AutoloadingTest.phpnu [ PK ! z " test/JsonMachineTest/ItemsTest.phpnu [ PK ! , ; C" test/JsonMachineTest/Exception/SyntaxErrorExceptionTest.phpnu [ PK ! xZ Z & $ test/JsonMachineTest/formatted-cr.jsonnu [ PK ! Z Z & t% test/JsonMachineTest/formatted-lf.jsonnu [ PK ! s5G G 7 $& test/JsonMachineTest/JsonDecoder/ExtJsonDecoderTest.phpnu [ PK ! ٕ 6 . test/JsonMachineTest/JsonDecoder/DecodingErrorTest.phpnu [ PK ! Ϋ 4 1 test/JsonMachineTest/JsonDecoder/ValidResultTest.phpnu [ PK ! N) ) 6 A4 test/JsonMachineTest/JsonDecoder/InvalidResultTest.phpnu [ PK ! _! = 6 test/JsonMachineTest/JsonDecoder/ErrorWrappingDecoderTest.phpnu [ PK ! B 8 A test/JsonMachineTest/JsonDecoder/PassThruDecoderTest.phpnu [ PK ! ʢ?c c 0 pC test/JsonMachineTest/JsonDecoder/StubDecoder.phpnu [ PK ! m?I I ) 3E test/JsonMachineTest/StringChunksTest.phpnu [ PK ! :F7 7 # H test/JsonMachineTest/TokensTest.phpnu [ PK ! # _i test/JsonMachineTest/ItemsTest.jsonnu [ PK ! q $ i test/performance/testPerformance.phpnu [ PK ! = ' q test/performance/twitter_example_1.jsonnu [ PK ! ($2 $2 ' test/performance/twitter_example_0.jsonnu [ PK G G - e